Essentia Spectrum
OUTPUTComputes the FFT from raw audio and outputs the raw result: spectrum (magnitude) and phase channels of fftSize/2+1 bins as a static sample buffer. For visualization, GLSL, resynthesis, or custom feature work. The analyzers run their own FFT and do not read from it.
- InputAudio CHOP (mono, first channel used)
- Output2 channels (
spectrum,phase) × (fftSize + zeroPad)/2 + 1 samples: 513 at the default FFT 1024. A static indexed buffer (sample = frequency bin), not time-domain. Bins are spaced sampleRate / (fftSize + zeroPad) apart. - ProcessingLatest fftSize samples each cook → Windowing (Hann / Hamming / Triangular / Blackman-Harris) → Zero Padding (optional) → FFT → CartesianToPolar (magnitude + phase)
Parameters
| Parameter | Type | Default | Options / Range | Description |
|---|---|---|---|---|
| FFT Size | Menu | 1024 | 512 / 1024 / 2048 / 4096 / 8192 / 16384 | Window size for FFT computation |
| Hop Size | Int | 512 | 64 – 16384 | No effect on the output: Spectrum analyzes the latest window once per cook. Reported on the info channels only |
| Window Type | Menu | Blackman-Harris 62 | Hann / Hamming / Triangular / Blackman-Harris 62/70/74/92 | Window function applied before FFT |
| Zero Padding | Menu | None | None / Half FFT / Full FFT | Appends zeros before the FFT: more, finer-spaced bins (769 or 1025 at FFT 1024), interpolated rather than truly sharper |
Larger FFT sizes improve frequency resolution (more bins, better at distinguishing close pitches) at the cost of time resolution (each frame covers more audio, smearing transients). This is exactly why each analyzer runs its own FFT: Tonal defaults to a large window (auto: 4096+), Rhythm to a short one (1024). For Spectrum itself the default 1024 is a good general-purpose balance; go bigger only when you need finer bins.
Each window trades main-lobe width for side-lobe suppression. The default Blackman-Harris 62 follows Essentia's recommended analysis configuration; Hann is the lighter classic alternative. The higher Blackman-Harris variants offer progressively stronger side-lobe suppression (70/74/92 dB) at the cost of wider main lobes.
Appends zeros to the windowed frame before FFT, which interpolates spectral bins without changing frequency resolution. This improves the accuracy of peak-based descriptors (centroid, rolloff, pitch) and produces smoother spectrum plots. “Half FFT” adds fftSize/2 zeros; “Full FFT” doubles the frame.
Output channels
| Channel | Range | What it does & how to use it |
|---|---|---|
spectrum | 0+ (fftSize/2+1 samples) | Raw FFT magnitude bins. Visualize as a bar graph, 3D terrain, or heatmap. Use as input for custom analysis. Bin 0 is DC and the last bin is Nyquist (sampleRate / 2); lower bins are bass, upper bins treble. Slice a range of samples with a Trim CHOP to isolate frequency bands. |
phase | −π to π (radians) | Phase of each bin. Pairs with spectrum for resynthesis, phase-vocoder style effects, or phase-aware custom features. Rarely useful to visualize directly. |
Essentia Spectral
ANALYSISComputes spectral shape descriptors and mel-frequency band energies, with optional PCA reduction of the feature set. Supports Realtime (per-frame, own FFT) and Batch (full-file offline analysis). Each feature group can be independently toggled.
- InputAudio / File In CHOP (raw audio, both modes)
- OutputRealtime: 1 sample per channel at frame rate | Batch: N samples (one per analysis frame). Channels:
mfcc0–mfcc12,spectral_centroid,spectral_flux,spectral_rolloff,spectral_contrast0–spectral_contrast5,hfc,spectral_complexity, mel bands (mel0_0_129style, band index then its low and high edge in Hz, with Mel Freq Names on; plainmel0–melNwith it off),pc0–pcN(if PCA enabled),pc_var0–pc_varN(if PCA Variance Channels enabled). Only enabled features produce channels. - FeaturesMFCC (timbral fingerprint, 13 coefficients) ·Centroid (brightness) ·Flux (rate of spectral change) ·Rolloff (85% energy frequency) ·Contrast (peak-valley difference in 6 bands) ·HFC (high-frequency content) ·Complexity (number of spectral peaks) ·Mel Bands (perceptual frequency band energies)
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Mode | Menu | Realtime | Realtime (per-frame from raw audio, own FFT) / Batch (full-file offline) |
| FFT Size | Menu | 2048 | Analysis window, 512 to 16384. Applies in both modes (page Analysis) |
| Hop Size (Batch) | Int | 1024 | Samples between analysis frames. Batch only (Realtime analyzes the latest window once per cook) |
| Zero Padding (Batch) | Menu | None | None / Half FFT / Full FFT. Batch only |
| Window Type | Menu | Blackman-Harris 62 | Window function. Applies in both modes |
| Compute (Batch) | Pulse | – | Run the batch analysis now. Only visible in Batch mode |
| Auto Compute (Batch) | Toggle | On | Re-run when the input audio changes (parameter changes only raise a stale warning). Only visible in Batch mode |
| Enable MFCC | Toggle | On | Enable/disable MFCC output channels |
| MFCC Count | Int | 13 | Number of MFCC coefficients (1–20) |
| MFCC Low Freq | Float | 0 Hz | Lower frequency bound for MFCC mel filters |
| MFCC High Freq | Float | 11000 Hz | Upper frequency bound for MFCC mel filters |
| Enable Centroid | Toggle | On | Enable spectral centroid |
| Enable Flux | Toggle | Off | Enable spectral flux |
| Flux Half Rectify | Toggle | Off | Only count energy increases (onset emphasis) |
| Flux Norm | Menu | L2 | L1 or L2 norm for difference computation |
| Enable Rolloff | Toggle | Off | Enable spectral rolloff |
| Rolloff Cutoff | Float | 0.85 | Energy fraction threshold (0.5 = median, 0.85 = standard, 0.95 = brightness) |
| Enable Contrast | Toggle | Off | Enable spectral contrast |
| Contrast Bands | Menu | 6 | Number of octave sub-bands (4 / 6 / 8) |
| Enable HFC | Toggle | On | Enable high-frequency content |
| HFC Type | Menu | Masri | Masri / Jensen / Brossier: different HFC formulations |
| Enable Complexity | Toggle | On | Enable spectral complexity |
| Complexity Threshold | Float | 0.005 | Minimum peak magnitude to count (0–0.1) |
| Enable Mel Bands | Toggle | On | Enable mel band output channels |
| Mel Bands Count | Menu | 40 | 24 / 40 / 60 / 80 / 128 |
| Mel Low Freq | Float | 0 Hz | Lower frequency bound for mel filters |
| Mel High Freq | Float | 22050 Hz | Upper frequency bound for mel filters |
| Mel Freq Names | Toggle | On | Include frequency ranges in channel names |
| Log Mel (dB Scale) | Toggle | Off | Convert mel band output to dB scale |
| PCA PAGE | |||
| Enable PCA | Toggle | Off | Enable PCA dimensionality reduction: projects all enabled spectral features into a compact set of principal components |
| PCA Components | Int | 3 | Number of principal components to output (2–16). Clamped to the number of enabled spectral channels |
| PCA Window Size (RT) | Menu | 512 | Circular buffer size for covariance estimation (128 / 256 / 512 / 1024 / 2048 / 4096). Larger = more stable, smaller = more responsive |
| PCA Update Rate (RT) | Int | 1 | How often to recompute eigenvectors per second (1–60 Hz). Lower values reduce CPU load |
| PCA Variance Channels | Toggle | Off | Append pc_var0–pc_varN channels showing the variance ratio explained by each component |
PCA reduces many correlated spectral features (MFCCs, mel bands, centroid, etc.) into a few decorrelated dimensions that capture the most variance. With 3 components you get a compact "spectral signature" suitable for driving 3D position, color (RGB), or clustering. Realtime mode uses a sliding window with throttled updates for stability; Batch mode computes PCA over the entire file. Sign-flip correction ensures temporal coherence between recomputations.
The default 0–11000 Hz covers the full speech/music range. For voice-only analysis, narrow to 80–3400 Hz to exclude sub-bass and high-frequency noise. For full-band analysis, set High Freq to the Nyquist (sampleRate/2).
Masri weights by energy×frequency (default), Jensen by amplitude×frequency² (stronger high-frequency emphasis), Brossier by amplitude×frequency (linear). Jensen and Brossier respond more aggressively to transients in the upper spectrum.
Output channels
| Channel | Range | What it does & how to use it |
|---|---|---|
mfcc0–mfcc12 | ~−50 to 50 | Timbral fingerprint. MFCCs capture the "texture" of a sound independent of pitch. Use them to cluster similar sounds, drive visual style transitions based on tonal character, or distinguish instruments. Feed into a Math CHOP for normalization, then map to shader uniforms. |
spectral_centroid | 0 – sr/2 Hz | Brightness. The "center of mass" of the spectrum. High values mean bright, shimmery sounds; low values mean warm, bassy tones. Map to color temperature (orange ↔ blue), particle speed, or lighting intensity. |
spectral_flux | 0+ | Rate of spectral change. Spikes when the timbre shifts suddenly: a new instrument enters, a filter sweeps, or a transition occurs. Use to trigger visual scene changes, glitch effects, or novelty highlights. |
spectral_rolloff | 0 – sr/2 Hz | Energy distribution edge. The frequency below which 85% of spectral energy lies. Distinguishes bright mixes from dark ones. Control cutoff-style visual filters, blur intensity, or fog density. |
spectral_contrast0–5 | unbounded | Per-band tonal vs. noisy content. Six sub-bands measuring peak-to-valley ratio. Drive multi-layer visual intensities: assign each band to a separate geometry layer or glow ring for a multi-band reactive sculpture. |
hfc | 0+ | High-frequency content. Sensitive to cymbals, hi-hats, and sibilance. Use for sparkle/shimmer particle effects, percussive high-end triggers, or treble-reactive lighting. |
spectral_complexity | 0+ | Number of spectral peaks. Simple tones (sine, flute) score low; complex sounds (orchestra, noise) score high. Drive visual density: particle count, fractal detail, or geometric subdivision level. |
mel0–melN | 0+ | Perceptual frequency band energies. Mel bands approximate human hearing: evenly spaced in perceived pitch, not linear frequency. Ideal for multi-band audio visualizers, frequency-mapped color gradients, per-band particle emitters, or as input features for ML classifiers. |
pc0–pcN | unbounded | Principal components (PCA). Reduces all enabled spectral features into a compact set of decorrelated dimensions ordered by variance. pc0 captures the dominant pattern of variation across your features. Map to position, color, or shape parameters for a low-dimensional "summary" of the audio's spectral character. Requires Enable PCA = On. |
pc_var0–pc_varN | 0 – 1 | Variance ratios. How much of the total feature variance each principal component explains. Use to auto-weight visual mappings: components with higher variance carry more information. Requires Enable PCA and PCA Variance Channels = On. |
Essentia Tonal
ANALYSISDetects pitch, harmonic content, musical key, dissonance, and inharmonicity. Supports Realtime (per-frame with EMA smoothing) and Batch (full-file with global or windowed key detection). FFT Size defaults to auto: semitone-safe at any sample rate.
- InputAudio / File In CHOP (raw audio, both modes)
- OutputRealtime: 1 sample per channel | Batch: N samples per channel. Channels:
pitch,pitch_confidence,note(with Enable Pitch Note), chromanote_a–note_gs,key,major_minor,key_strength,dissonance,inharmonicity. With Musical Labels off the names becomepitch_note,hpcp0–hpcpNandkey_scale. Only enabled features produce channels. - FeaturesPitch (YinFFT algorithm, Hz + confidence) ·HPCP (harmonic pitch class profile / chroma, 12/24/36 bins) ·Key (key + major/minor + strength) ·Dissonance (sensory roughness) ·Inharmonicity (deviation from harmonic series)
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Mode | Menu | Realtime | Realtime (per-frame from raw audio, own FFT) / Batch (full-file offline) |
| FFT Size | Menu | auto | Both modes. Auto picks the smallest semitone-safe window for the incoming sample rate (4096 at 44.1/48 kHz, 8192 at 88.2/96 kHz); explicit smaller sizes warn |
| Hop Size (Batch) | Int | 2048 | Samples between analysis frames. Batch only |
| Window Type | Menu | Blackman-Harris 62 | Window function. Applies in both modes |
| Zero Padding (Batch) | Menu | None | None / Half FFT / Full FFT. Batch only |
| Compute (Batch) | Pulse | – | Trigger batch analysis |
| Auto Compute (Batch) | Toggle | On | Re-run when the input audio changes (parameter changes only raise a stale warning) |
| Pitch Algorithm | Menu | YinFFT | YinFFT |
| HPCP Size | Menu | 12 | 12 (default, 1 per semitone) / 24 / 36 bins |
| Enable Pitch | Toggle | On | Enable pitch detection |
| Pitch Min Freq | Float | 20 Hz | Minimum detectable frequency (constrain to instrument range) |
| Pitch Max Freq | Float | 22050 Hz | Maximum detectable frequency |
| Pitch Tolerance | Float | 1.0 | Peak detection strictness (lower = fewer octave errors, more unvoiced frames) |
| Enable HPCP | Toggle | On | Enable chroma output |
| HPCP Harmonics | Int | 0 | Harmonic contributions to the chroma channels (0–12; 0 = fundamental only, 3–5 for harmonic instruments) |
| Reference Freq | Float | 440 Hz | Tuning reference, 415–466 Hz (415 = Baroque, 432 = alternative, 440 = standard) |
| HPCP Non-Linear | Toggle | Off | Apply peak-sharpening post-processing |
| HPCP Normalized | Menu | Unit Max | Unit Max / Unit Sum / None |
| Enable Key | Toggle | On | Enable key detection |
| Key Frames (RT) | Int | 8 | HPCP frames to average for key detection (1–300). Realtime only |
| Key Profile | Menu | Bgate | Bgate (default) / Temperley / Krumhansl / EDMA / Diatonic / Gomez |
| Peak Threshold | Float | 0.00001 | Minimum spectral peak magnitude. Filters noise-floor peaks from HPCP/Key/Dissonance/Inharmonicity |
| Peak Max Freq | Float | 3500 Hz | Upper frequency limit for spectral peak detection; tonal content lives below 3500 Hz |
| Enable Dissonance | Toggle | On | Enable dissonance output |
| Enable Inharmonicity | Toggle | On | Enable inharmonicity output |
| Musical Labels | Toggle | On | Use note names (A through G#) instead of indices for HPCP channels |
| Enable Pitch Note | Toggle | Off | Output pitch-to-note-class channel |
| Smoothing (RT) | Float | 0.5 | EMA smoothing coefficient (0 = none, 0.99 = maximum). Realtime only; does not affect key output |
| Key Mode (Batch) | Menu | Global | Global (single key for whole file) / Windowed (per-frame key). Batch only |
| Key Window Size (Batch) | Int | 8 | HPCP frames for windowed key detection (1–300). Batch only, visible when Key Mode = Windowed |
The defaults follow Essentia’s MusicExtractor recommendations. Here are per-use-case tweaks:
FFT Size is a correctness setting for key, not a latency knob. The spectrum has to separate adjacent semitones, so the requirement scales with sample rate: 4096 at 44.1/48 kHz, 8192 at 96 kHz. Auto picks the right one. Below it the key goes wrong with a confident-looking strength (measured on one track at otherwise identical settings: 4096 correct, 2048 a fifth away, 1024 elsewhere again, in both Realtime and Batch). The operator warns when you cross that line, so if you see “Spectrum too coarse for key/HPCP”, raise FFT Size rather than trusting the reading.
Key Detection (best accuracy): the defaults are already the accurate setting. The key stage runs its own chroma pinned to Essentia’s KeyExtractor configuration, so HPCP Size, HPCP Harmonics and HPCP Non-Linear do not affect the key output. Worth tuning: Key Profile (Bgate by default, matching Essentia), Key Frames = 30–60 in Realtime so the window spans whole chords rather than a fraction of one. For batch, use Global key mode unless tracking modulations.
Pitch Tracking: Pitch Algorithm = YinFFT (default). Constrain frequency range to your source: voice 80–800 Hz, guitar 80–1200 Hz, bass 30–300 Hz. Smoothing = 0.3 for responsiveness.
Chord / Harmony Analysis: HPCP Size = 36, HPCP Harmonics = 8, HPCP Non-Linear = On, Key Profile = EDMA (electronic/dance) or Temperley (pop/rock).
Live Visuals (fast RT): HPCP Size = 12 (default), Musical Labels = On (default), Smoothing = 0.5, Key Frames = 8 (default). Enable Pitch Note for a 0–11 note class output.
Different profiles are tuned for different genres. Bgate (default) is Essentia’s own default and the profile its KeyExtractor uses. Temperley is a well-tested general-purpose alternative. Krumhansl is the classical music theory standard. EDMA is designed for electronic/dance music. Diatonic is the simplest model. Gomez is optimized for guitar-heavy material.
Constraining to instrument-appropriate bands eliminates octave errors. Common ranges: guitar 80–1200 Hz, voice 80–800 Hz, bass 30–300 Hz.
When set to 0 (default), only the fundamental contributes to chroma. Setting to 3–5 makes HPCP more robust for harmonic instruments (piano, guitar, voice) where overtones reinforce the pitch class. This affects the chroma output channels (note_* / hpcpN) only. Key detection is unaffected: it runs its own chroma fixed at 4 harmonics, matching Essentia’s KeyExtractor.
The default threshold (0.00001) and max frequency (3500 Hz) match Essentia’s MusicExtractor. These filter noise-floor peaks and limit analysis to the tonal range where pitch classes are meaningful. Increase threshold for noisier signals; raise max frequency above 3500 only for very high-pitched instruments.
Output channels
| Channel | Range | What it does & how to use it |
|---|---|---|
pitch | 0+ Hz | Fundamental frequency. Detected via YinFFT. Map to note names for generative music notation, control animation speed or vertical position with pitch, or track vocals for lip-sync effects. |
pitch_confidence | 0 – 1 | Pitch reliability. Low during noise or silence, high for clear tonal content. Use as a gate: only apply pitch-driven effects when confidence exceeds a threshold. Crossfade between pitched and unpitched visual modes. |
note | 0 – 11, or −1 | Pitch as a note class. Needs Enable Pitch Note. The detected pitch rounded to the nearest semitone: 0 = C, 1 = C#, ... 9 = A, ... 11 = B, relative to Reference Freq. Reads −1 when no pitch is detected. Named pitch_note with Musical Labels off. Drive note-indexed palettes, sample triggers, or per-note geometry. |
note_a–note_gs | 0 – 1 | Chroma / pitch class energy. 12 bins (one per semitone, A through G#) with Musical Labels on (default). Bin 0 = A (reference frequency). Visualize as a harmony wheel, map each note to a unique color, detect chord changes for scene transitions, or build a real-time piano-roll display. HPCP Size 24 or 36 adds sub-semitone bins (note_a, note_a+, ... or note_a, note_a~, note_a+, ...). With Musical Labels off the channels are hpcp0–hpcpN. |
key | 0 – 11 | Musical key. Encoded integer (0=C, 1=C#, ... 11=B). Assign color palettes or scene themes per key. Drive generative art parameters that shift as the music modulates to a new key. |
major_minor | 0 or 1 | Major or minor. 0 = major, 1 = minor. Use for mood: major keys → warm, bright, expansive palettes; minor keys → cool, dark, constrained palettes. Switch between two visual presets. |
key_strength | 0 – 1 | Key confidence. Higher when the harmonic content strongly matches a key template. Blend strength into color saturation or confidence-gated transitions. |
dissonance | 0 – 1 | Sensory roughness. Measures perceptual dissonance between spectral peaks. Map to visual chaos: glitch intensity, distortion amount, turbulence in fluid simulations, or camera shake. |
inharmonicity | 0 – 1 | Deviation from harmonic series. Low for flutes and voices, high for bells and percussion. Distinguish melodic from percussive sources, or drive material textures (metallic vs. organic). |
Essentia Rhythm
ANALYSISDetects onsets and estimates tempo, with optional kick / perc / hat drum triggers. Realtime analyzes its own short window (default 1024) with TempoTapDegara BPM and Onsets-style adaptive thresholding. Batch uses RhythmExtractor2013 for full-file BPM and beat tracking.
- InputAudio / File In CHOP (raw audio, both modes)
- OutputRealtime: 1 sample per channel | Batch: N samples per channel. 6 channels:
onset(0/1 trigger),onset_strength,bpm,beat(0/1 trigger),beat_phase(0–1 sawtooth),beat_confidence. With Enable Drums on, 6 more:low,mid,high(band levels 0–1) andkick,perc,hat(0/1 triggers) - ProcessingRT: OnsetDetection/SuperFlux → Onsets-style adaptive threshold → onset trigger. ODF history → TempoTapDegara → BPM + tick-anchored beat phase. Batch: RhythmExtractor2013 → BPM/beats + per-frame onset detection via Onsets algorithm.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Mode | Menu | Realtime | Realtime (per-frame from raw audio, own FFT) / Batch (full-file offline) |
| Window Size (RT) | Menu | 1024 | Realtime analysis window (512 / 1024 / 2048 / 4096). 1024 is the measured best default; the operator warns if the window is shorter than the per-cook audio chunk |
| Window Type | Menu | Blackman-Harris 62 | Window function. Applies in both modes |
| FFT Size (Batch) | Menu | 2048 | Window size for the batch FFT (512 to 16384). Batch only; Realtime uses Window Size |
| Hop Size (Batch) | Int | 512 | Samples between analysis frames (~11.6 ms at 44100 Hz). Batch only |
| Zero Padding (Batch) | Menu | None | None / Half FFT / Full FFT. Batch only |
| Compute (Batch) | Pulse | – | Trigger batch analysis |
| Auto Compute (Batch) | Toggle | On | Re-run when the input audio changes (parameter changes only raise a stale warning) |
| Onset Method | Menu | Complex | HFC / Complex / Flux / Mel Flux / RMS / SuperFlux |
| Onset Sensitivity | Float | 0.5 | 0.0 (rare triggers) – 1.0 (frequent). In batch mode, maps to Onsets algorithm alpha and silence threshold |
| BPM Min / Max | Int | 60 / 180 | BPM search range (internally clamped to [40,180] / [60,250] for Essentia algorithms) |
| Rhythm Method (Batch) | Menu | Degara | Degara / Multi-Feature: RhythmExtractor2013 algorithm variant, Batch only |
| DRUMS PAGE | |||
| Enable Drums | Toggle | Off | Adds low mid high kick perc hat after the rhythm channels, both modes. The rhythm channels are bit-identical either way |
| Drum Window | Menu | 2048 | 1024 / 2048: the drum detector's own FFT, independent of Window Size and FFT Size. 2048 catches kicks better, 1024 reacts up to 10 ms sooner |
| Kick / Perc / Hat Sensitivity | Float | 0.5 | 0 – 1 per class. Higher fires more often; at 0 only a hit louder than anything recent fires |
Each band fires on a jump in its own spectral flux relative to that band's recent hits (perc also has to clear twice its average over the last 0.1 s), with a 50 ms refractory and a level gate so near-silence never fires. It does not know what a drum sounds like: on a full mix a bass line can fire kick and a guitar or vocal can fire perc.
Measured on the held-out half of a 171-track drum corpus (F-measure at ±50 ms), Realtime at 60 fps with the defaults: kick 0.835, perc 0.632 (0.443 on full band mixes), hat 0.813; median latency 27 / 23 / 19 ms. Batch at Hop Size 512: 0.882 / 0.645 / 0.825, and within 0.002 of that at Hop Size 1024 or 2048. Cost: about 0.05 ms per cook.
It adapts to the loudest recent hits: after switching to a much quieter source give it up to half a minute, or toggle Enable Drums. In Batch, drum triggers land near each hit's true time while onset fires about FFT Size / 2 early (29 ms at the defaults), so allow for that offset when combining them. The Info CHOP channel drum_resyncs counts restarts after cooks the operator missed (Mode on Batch, nothing pulling it); it stays 0 in normal playback.
Complex (default) uses both magnitude and phase for the most accurate general-purpose onset detection. HFC emphasizes high-frequency transients, good for percussive material. Flux measures overall spectral change. Mel Flux applies mel-weighted spectral difference, more robust for harmonic/melodic content. RMS uses simple energy change: fast and reliable for broadband signals. SuperFlux uses TriangularBands + SuperFluxNovelty for music with soft/gradual onsets.
Realtime: BPM estimation uses Essentia’s TempoTapDegara algorithm, run periodically (~1.5 s) on accumulated onset detection history. BPM is derived from median tick intervals; beat phase is anchored to tick positions for audio-synchronized animation. The tempo clock runs on elapsed audio rather than on cook count, so BPM stays correct when TouchDesigner drops frames. Once the onset history is fully silent (a paused transport, digital silence), bpm holds its last value and beat_confidence drops to 0 instead of inventing a tempo. Batch: RhythmExtractor2013, which requires 44.1 kHz input; non-44.1 kHz audio is resampled internally (see the sample-rate callout below).
Realtime: Onset Method = Complex, Window Size = 1024 (both defaults; 1024 was validated by a measured onset-F1 window sweep). Batch: Onset Method = Complex, Rhythm Method = Degara, FFT 2048, Hop 512 (all defaults). For both modes: set the narrowest BPM range that covers your material (e.g., 100–140 for house, 80–160 for pop). Use SuperFlux for music with soft/gradual onsets.
RhythmExtractor2013 requires 44.1 kHz input. Non-44.1 kHz audio is resampled internally (libsamplerate) before BPM extraction, verified against a native 44.1 kHz analysis of the same material, agreeing to two decimals. Onset detection is frame-based and rate-aware, so it never needed resampling. If you have batch BPM numbers from v1.1.8: that release shipped without the resampler, so BPM on non-44.1 kHz files came out scaled by roughly rate/44100 (about 9% high on 48 kHz files). Expect v2.0 to report different, correct values on the same files.
Realtime Rhythm analyzes the latest Window Size samples once per cook (512–4096, default 1024). Two measured facts guide the choice: 1024 is the validated default (onset F1 0.461 on a 262-onset reference; 4096 collapses to 0.028 because long windows smear onset timing), and if the per-cook audio chunk exceeds the window, the excess is never analyzed: at 30 fps / 48 kHz a 1024 window skips 36% of the audio and detection degrades sharply. The operator warns whenever window < sampleRate / fps and the fix is either direction: raise Window Size or raise TD’s fps.
Output channels
| Channel | Range | What it does & how to use it |
|---|---|---|
onset | 0 or 1 | Transient trigger. Fires on detected attacks: drum hits, plucks, consonants. Use for particle bursts, flash/strobe triggers, camera cuts, step-sequenced events, or instantiating geometry. |
onset_strength | 0+ | Continuous onset function. The raw detection signal before thresholding. Scale burst intensity or particle count by strength for velocity-sensitive triggers. Smooth with a Lag CHOP for an onset envelope. |
bpm | BPM min – max | Estimated tempo. Sync LFO rates, animation cycle durations, or generative pattern timing to the music. Divide by 60 to get beats-per-second for direct use in Speed parameters. |
beat | 0 or 1 | Beat trigger. Fires on the estimated beat grid. Use for rhythmic pulsing, beat-locked scene transitions, synchronized step-sequencing, or quantized color cycling. |
beat_phase | 0 – 1 (sawtooth) | Position within beat. Ramps linearly from 0 to 1 between beats. Feed into easing curves (Lookup CHOP) for smooth beat-synced animation: bouncing, breathing, pendulum swings, or rhythmic camera motion. |
beat_confidence | 0 – 1 | Beat tracking reliability. Low during ambient or arrhythmic passages, high during steady beats. Use to fade in beat-synced effects only when tracking is stable, or crossfade to an onset-only mode when confidence drops. |
low mid high | 0 – 1 | Drum band levels (Enable Drums). Energy in 30–150 Hz, 200–2500 Hz and 10–20 kHz, each relative to its own recent peak, so they stay usable across quiet and loud material. Drive bass-heavy scaling, mid-band pulses or a high-end shimmer layer without building your own band filters. |
kick | 0 or 1 | Kick trigger (Enable Drums). Fires for one cook on a jump in the low band. Camera kicks, floor pulses, bass-drop flashes. On full mixes a bass line can fire it too. |
perc | 0 or 1 | Mid-band hit trigger (Enable Drums). Snares, claps, rims and toms. Backbeat accents, strobes on 2 and 4. The least reliable of the three on full mixes, where guitars and vocals share the band. |
hat | 0 or 1 | High-band hit trigger (Enable Drums). Hi-hats, shakers, cymbals. Sparkles, particle spawns, fast rhythmic detail. |
Essentia Loudness
METERINGEBU R128 loudness metering with momentary, short-term, and integrated measurements, plus RMS energy and zero-crossing rate. Supports Realtime (per-frame via ring buffer) and Batch (full-file offline analysis).
- InputAudio / File In CHOP (raw audio, both modes)
- OutputRealtime: 1 sample per channel | Batch: N samples per channel. 7 channels:
loudness(instantaneous dB),loudness_momentary,loudness_shortterm,loudness_integrated,dynamic_range,rms,zcr - ProcessingAudio → ring buffer → frame dispatch → Essentia Loudness → dB conversion → sliding windows (momentary 400ms, short-term 3s) → EBU R128 two-pass gating for integrated. RMS and ZCR computed directly from the audio frame.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Mode | Menu | Realtime | Realtime (per-frame via ring buffer) / Batch (full-file offline) |
| Compute (Batch) | Pulse | – | Trigger batch analysis |
| Auto Compute (Batch) | Toggle | On | Re-run when the input audio changes (parameter changes only raise a stale warning) |
| Frame Size | Menu | 1024 | 512 / 1024 / 2048 |
| ZCR Threshold | Float | 0 | Dead-band around zero for ZCR (0–0.1). Increase to filter noise-floor chatter on quiet signals |
Output channels
| Channel | Range | What it does & how to use it |
|---|---|---|
loudness | dB (−144 floor to ~0) | Instantaneous perceived loudness. The fastest-responding level signal. Use for frame-level reactive scaling: VU-meter needle, geometry size, or opacity. Responds immediately to transients. |
loudness_momentary | dB | EBU R128 momentary (400 ms). A smoothed level that tracks phrase-level dynamics without jitter. Use for dynamic scaling of visual elements, responsive but stable brightness, or gain-riding effects. |
loudness_shortterm | dB | EBU R128 short-term (3 s). Captures the current "energy zone" of the track. Use for scene-level intensity: ambient lighting adjustments, macro-level background color, or fog density that follows the song's structure. |
loudness_integrated | dB | Running gated average. EBU R128 integrated loudness with absolute + relative gating. In Batch it covers the file up to each frame. In Realtime it is a session meter: it accumulates everything heard since the operator last reconfigured (changing Frame Size resets it), so compare it to a per-file LUFS figure only after a reset. Use for overall show level monitoring, normalization reference, or auto-gain to keep visuals consistent across tracks of different loudness. |
dynamic_range | dB (0+) | Loudness swing over 3 s. Peak-to-valley of the short-term window. Detects builds and drops: high values mean the audio is moving between quiet and loud. Drive contrast-based transitions, tension/release mapping, or drop-triggered explosions. |
rms | 0 – 1 | Root mean square energy. The classic "audio reactive" signal: simple, linear amplitude. Map directly to geometry scale, opacity, displacement amount, or any parameter that should pulse with the music's energy. |
zcr | 0 – 1 | Zero-crossing rate. Measures how "noisy" vs. "tonal" the audio is. High ZCR = noise, percussion, sibilance. Low ZCR = clean pitched tones. Use to drive grain/static effects, distinguish drums from melody, or control visual roughness. |