EssentiaTD

Essentia CHOP suite

EssentiaTD

Real-time and batch audio analysis for TouchDesigner.

Five C++ CHOP plugins powered by Essentia: spectrum, mel bands, MFCCs, pitch, key, onsets and BPM, drum triggers and EBU R128 loudness, running natively inside TouchDesigner. Free and open source.

v2.1.0 · free and open source (AGPL-3.0) · Windows x64 and macOS

EssentiaTD v2.1.0Powered by Essentia · Darien Brito
Demo

See it run

Operators

Five operators

Every operator takes raw audio. The four analyzers have a Mode parameter that switches between real-time and batch analysis, and each runs its own FFT internally in both modes.

Full parameters, defaults and every output channel for each operator: the reference.

Two modes

Real-time or a whole file

Realtime

Wire an Audio CHOP straight into any analyzer. Incoming audio accumulates in an internal ring buffer; each cook the operator windows and FFTs the latest full analysis window and outputs 1 sample per channel at TD's cook rate. Each operator uses the window its algorithm wants (Tonal resolves semitones, Rhythm keeps transients sharp) with no shared FFT node to configure.

Batch mode

Set Mode to Batch and the operator processes an entire audio buffer (e.g. from an Audio File In CHOP) on a background thread. Output contains one sample per analysis frame across the full file. Auto Compute is on by default, so the analysis re-runs whenever the input audio changes; turn it off and press Compute to run on demand. Auto Compute follows the audio only: change an analysis parameter and the cached result stays on display, with a warning that it is stale until you press Compute. Flipping Mode never requires rewiring: both modes read the same audio input.

System

Requirements

Windowsx64. The installer puts the plugins in your user folder, no admin rights.
macOSApple Silicon and Intel (universal build).
TouchDesignerLoads the five operators as custom CHOPs from its Plugins folder.
InputMono: the operators analyze one channel. Average stereo to mono first (see Mono by design).
Get it

Get EssentiaTD

Free and open source under AGPL-3.0. The download links always point to the latest release.

06-15 GuideEssentiaTD v2.1.0

Using EssentiaTD.

Installing, how the operators connect, FFT settings, drum triggers, real-time timing and the details that change your numbers.

Setup

Installing

Close TouchDesigner, run the installer for your platform, then start TouchDesigner again. The operators appear in the OP Create Dialog under CHOP, named Essentia Spectrum, Essentia Spectral and so on.

Installs to
WindowsDocuments\Derivative\Plugins\Essentia for the current user. The installer checks that TouchDesigner is closed.
macOS~/Library/Application Support/Derivative/TouchDesigner099/Plugins/Essentia. Installed from the pkg, the plugins load without per-bundle Gatekeeper prompts.
The installers are unsigned. Windows: SmartScreen may warn about an unrecognized publisher: click More info, then Run anyway. macOS: if the pkg will not open, go to System Settings, Privacy & Security, and click Open Anyway (on older macOS, Control-click the pkg and choose Open).

Prefer copying files yourself? Download the zip for your platform from the releases page and copy the five plugins (.dll on Windows, .plugin on macOS) into the same folder. TouchDesigner scans subfolders. Unsigned plugins from a zip ask for permission once each on first load.

Building from source: see Build from source in the repository.

Topology

Signal flow

One topology for both modes: audio wires straight into every operator, and each runs its own FFT internally with the window its algorithm wants. There is no shared FFT node and no ordering constraint between analyzers.

Why per-operator FFT

A shared spectrum forces one FFT size onto operators with conflicting needs: Tonal needs 4096 to resolve semitones, but feeding Rhythm a 4096 spectrum instead of 1024 cut onset-detection F1 from 0.47 to 0.13 on a measured 263-onset reference (long windows blunt the transients onset detection keys on). v2.0 removes the conflict: each operator gets the window its algorithm wants, on the same audio wire. The converted Rhythm at window 1024 reproduces the v1.x implementation's onset F1 exactly.

Migrating from v1.x

A Spectrum CHOP wired into Spectral, Tonal, or Rhythm raises an error in both modes ("As of v2.0 this operator analyzes raw audio and computes its own FFT. Connect the audio directly"). Delete the Spectrum node from the analysis chain and wire the audio straight in. If you tuned an upstream Spectrum's FFT Size for Tonal, set Tonal's own FFT Size instead (or leave it on auto). Essentia Spectrum remains available as a raw magnitude + phase output.

Notes

FFT settings live on each operator

Each analyzer's Analysis page carries its FFT parameters. FFT Size and Window Type apply in both modes; Hop Size and Zero Padding are batch-only (realtime analyzes the latest window once per cook, and zero padding narrows bin spacing without adding true resolution).

  • Spectral defaults to 2048.
  • Tonal defaults to auto: the smallest FFT whose bin spacing separates adjacent semitones at the incoming sample rate (4096 at 44.1/48 kHz, 8192 at 88.2/96 kHz). Picking a smaller size explicitly raises a warning in both modes.
  • Rhythm batch uses FFT 2048 / hop 512; realtime has its own Window Size menu (512-4096, default 1024, see below).
Notes

Tonal and FFT resolution

Key and HPCP need bin spacing of at most 15.56 Hz (a semitone at middle C), which is why Tonal's auto default resolves to 4096 or 8192 depending on sample rate. Two things are worth knowing about the too-coarse warning:

It means "may be unreliable", not "is wrong". Measured across three tracks, the smallest adequate FFT size was content-dependent: 1024, 2048 and 4096 respectively. 4096 is simply the smallest size at which all three were correct.

Do not judge tonal output by how confident it looks. The wrong answer is not noisy or obviously broken; it presents as a stable, settled key. On one test track a 1024-point analysis produced the wrong key at strength 0.608 while the correct answer at 4096 reported only 0.531. key_strength cannot be used to detect this failure.

Full method and measurements: docs/tonal-fft-resolution.md.

Rhythm

Drum triggers

The Rhythm CHOP's Drums page adds six channels after the rhythm channels when Enable Drums is on, in both modes:

channel band what it is
low, mid, high 30-150 Hz, 200-2500 Hz, 10-20 kHz band energy, 0 to 1 relative to its own recent peak
kick, perc, hat the same three bands 1 for one cook when a hit is detected, otherwise 0

Names follow the palette's audioAnalysis; its snare is perc here, because the mid band fires on snares, claps, rims and toms alike. Drum Window (1024 / 2048, default 2048) is the detector's own FFT, independent of Window Size and FFT Size. Kick / Perc / Hat Sensitivity (0-1, default 0.5): higher fires more often; at 0 only a hit louder than anything recent fires.

Each band fires on a jump in its own log-compressed spectral flux above a fraction of that band's running peak (perc also above twice its mean over the last 0.1 s), with a 50 ms refractory and a level gate around -80 dBFS so near-silence never triggers. It does not know what a drum sounds like: on a full mix a bass line can fire kick and a guitar or vocal can fire perc.

Measured on the held-out half of a 171-track drum corpus (F-measure at +/-50 ms), Realtime at 60 fps with the defaults: kick 0.835, perc 0.632 (scored on snares; 0.443 on full band mixes), hat 0.813, median latency 26.7 / 23.1 / 19.3 ms. Batch (Hop Size 512): 0.882 / 0.645 / 0.825. Batch drums step at 512 samples or finer whatever the Hop Size, and land close to each hit's true time. The batch onset channel does not: its frame f covers the audio from f x Hop Size onward, so it fires about FFT Size / 2 early (measured 29 ms early at the defaults, against -3 ms for kick). Keep that offset in mind when combining onset with the drum triggers in Batch. Drums on costs about 0.05 ms per cook. The six rhythm channels are bit-identical with drums on or off.

If the operator misses cooks (Mode on Batch, nothing pulling it, a rewired input), the drum window refills instead of joining old audio to new, and the Info CHOP channel drum_resyncs counts those restarts.

Rhythm

Analysis window vs cook rate

Realtime analysis reads the latest window of audio once per cook. Two consequences worth knowing:

If the per-cook audio chunk is larger than the analysis window, the excess is never analyzed. At 30 fps and 48 kHz each cook delivers 1600 samples; a 1024 window skips 576 of them (36% of the audio), and onset detection degrades sharply: measured onset F1 collapsed from 0.461 to below 0.16 across all windows at 30 fps. The operator warns whenever window < sampleRate / fps and tells you which way to fix it (raise the window, or raise TD's fps). Note this also fires for a 512 window at 60 fps (800-sample cooks): 512 is only fully sound above ~94 fps.

Longer windows are not safer. Onset timing smears with window length; at 4096 the onsets that fire are mostly mistimed beyond a 50 ms tolerance (precision 0.056). Keep the default 1024 unless you have a specific reason.

Rhythm

Batch BPM and sample rate

Batch BPM uses Essentia's RhythmExtractor2013, which requires 44.1 kHz input. Non-44.1 kHz audio is resampled internally (libsamplerate) before BPM extraction; onset detection is frame-based and rate-aware, so it never needed resampling. Verified: the same 48 kHz file analyzed natively at 44.1 kHz and through the resampler agree to two decimals.

If you have batch BPM numbers from v1.1.8: that release shipped without the internal resampler, so batch BPM on non-44.1 kHz files was computed at the native rate and came out scaled by roughly rate/44100 (about 9% high on 48 kHz files), with an easy-to-miss "Resample to 44100 Hz failed" warning. Expect v2.0 to report different (correct) values on the same files.

Spectrum

Raw data, not a display

Essentia Spectrum outputs a linear-bin FFT magnitude spectrum plus phase. Linear bins are what analysis and DSP code expect, but they look bottom-heavy when plotted directly; most musical detail is crammed into the lower bins. Use it for GLSL, resynthesis, or custom feature work. For a perceptually scaled on-screen spectrum, TouchDesigner's built-in Audio Spectrum CHOP is the better display choice.

Notes

Mono by Design

The suite processes a single audio channel. This is intentional: stereo analysis would double every output channel (e.g., mfcc0_L, mfcc0_R, spectral_centroid_L, spectral_centroid_R), making the output unwieldy and harder to map in a visual context.

If you need stereo-aware analysis, select each channel independently using a Select CHOP and run two separate analysis chains. This keeps the output organized and lets you choose which features to extract per channel.

Recommended approach for stereo sources. In most audio-reactive scenarios, collapsing to mono before analysis preserves all relevant information. Sum left and right with a Math CHOP (Combine Channels = Average) before feeding the analyzers. This captures the full frequency content of both channels without phase cancellation artifacts that a simple channel pick might miss.

Channel choice measurably shifts results. Batch mode analyzes channel 0 (left) only when fed a stereo CHOP. It warns, but the numbers still differ from a mono average. Measured on the same file with identical settings, Tonal's batch key_strength reads 0.825 from the L+R average and 0.882 from the left channel alone (same detected key). When comparing analysis values across sessions or against published numbers, always state how the audio was collapsed to mono.

Errors

Read the warnings

The operators report problems on the node instead of silently producing wrong numbers: an FFT too coarse for key detection, an analysis window shorter than the audio delivered per cook, a multichannel input (only channel 0 is analyzed), or batch results that are stale after a parameter change. In Realtime, if Spectral or Tonal rejects a parameter value for one feature (for example more mel bands than the FFT has bins), only that feature switches off and the warning carries Essentia's own reason; the operator errors only when no enabled feature can run.