Aether
02ProtocolAether v1.4.3

The wire format, open.

Aether streams on an open, MIT-licensed shared memory format. Any software that can map shared memory can read it. This is the full spec, for writing your own reader.

VST3 · AU · TD CHOPDarien Brito

Lock-free, single-producer / single-consumer (SPSC) protocol for streaming float32 audio + DAW transport between processes over OS shared memory. No daemon, no sockets, no kernel round-trips on the data path.

License: MIT. This specification may be freely implemented, copied, and redistributed. MIT-licensed reference implementations (C++ header-only and Python) live alongside this file in the Aether source tree and are available from the author.

  • Byte order: little-endian. Target archs: x86-64, ARM64.
  • All offsets are from the start of the shared memory segment.
  • u32/u64/i32/f32/f64 = fixed-width little-endian scalars.
  • Atomic fields require lock-free 64-bit atomics (C11 stdatomic, C++ std::atomic, or Interlocked*).

1. Naming

Object Windows POSIX (macOS/Linux)
Audio segment ShmAudio_<stream> (CreateFileMapping) /ShmAudio_<stream> (shm_open)
Ready event ShmAudio_<stream>_Ready (manual-reset event) none, poll shm_open
Registry segment ShmAudioRegistry /ShmAudioRegistry
Registry lock Local\ShmAudioRegistryLock (mutex) /ShmAudioRegistryLk (sem_open, init 1)

<stream> = user-chosen stream name, ≤63 bytes, null-terminated in headers.

POSIX name clamp: macOS limits shm names to 31 chars (PSHMNAMLEN). Names are truncated deterministically to 31 chars after prefixing, so producer and consumer must apply the same truncation. Consequence: stream names sharing a 21-char prefix collide on macOS.

2. Audio segment layout

Total size = 4096 + ringCapacity × numChannels × 4 bytes.

Offset Size Region
0 256 Header (§3), written once by producer
256 128 Transport state (§4), seqlock
384 256 Ring metadata (§5), atomics, one per cache line
640 3456 reserved (zero)
4096 capacity×channels×4 Audio data (§5), planar float32, page-aligned

3. Header (256 bytes, offset 0)

Written by the producer before the segment name/ready event becomes visible; read-only for consumers.

Offset Type Field Notes
0 u32 magic 0xA0D10500
4 u32 protocolVersion 1
8 u32 headerSize 256
12 u32 totalSegmentSize bytes
16 f64 sampleRate e.g. 48000.0
24 u64 createdTimestamp opaque, platform clock (§7)
32 u32 maxBlockSize max frames per producer push
36 u32 numChannels 1–16
40 u32 sampleFormat 0 = float32 (only format in v1)
44 u32 ringCapacity frames, power of two, 2048–65536
48 u32 audioDataOffset 4096
52 u32 transportOffset 256
56 u32 ringMetaOffset 384
60 u32 producerPid for liveness (§7)
64 char[64] streamName null-terminated
128 char[32] producerName null-terminated, informational
160 u64 producerStartToken producer process start stamp, 0 = unknown (§7)
168 u8[88] reserved zero

Validation on open (reject segment otherwise): magic, version, and headerSize match; ringCapacity is a power of two within §8's limits; 0 < numChannels ≤ 16; 0 < sampleRate ≤ 768000; maxBlockSize ≤ 65536 (0 is legal and means unknown); the three offset fields equal the §2 constants; totalSegmentSize == 4096 + capacity×channels×4. Then check the size of the object you actually mapped (VirtualQuery / fstat) and reject anything shorter than totalSegmentSize: a header that outruns its segment is truncated or hostile, and every pointer below is derived from the §2 constants these fields must match. v1 fixes the layout at those constants; moving it is a version bump, so a reader may hardcode them and validate the fields against them, which is what the reference readers do.

4. Transport state (128 bytes, seqlock)

Producer publishes DAW transport once per audio block. Guarded by a seqlock: odd sequence = write in progress.

Offset Type Field
0 u32 atomic sequence
4 u32 pad
8 f64 bpm
16 f64 samplePos (samples from project start)
24 f64 ppqPos (quarter notes)
32 f64 lastBarPpq
40 i32 timeSigNumerator
44 i32 timeSigDenominator
48 u32 transportFlags: bit0 playing, bit1 recording (reserved: the Aether producer never sets it), bit2 looping
52 u32 pad
56 u64 hostTimestamp (opaque, platform clock)
64 u8[64] reserved

Writer (producer only):

  1. sequence.store(seq+1, relaxed); atomic_thread_fence(release)
  2. write payload
  3. sequence.store(seq+2, release)

The release fence after the odd store is required. A plain release store only orders writes that precede it, not the payload writes that follow.

Reader:

  1. s1 = sequence.load(acquire); if odd, retry
  2. copy payload
  3. atomic_thread_fence(acquire); s2 = sequence.load(acquire)
  4. s1 == s2 → snapshot is consistent, else retry (bounded, e.g. 64 attempts)

5. Audio ring buffer

Data layout is planar: channel c occupies audioData[c*capacity .. c*capacity + capacity-1] (float32 frames). This lets both sides use straight memcpy per channel.

Ring metadata (offset 384; each OWNER starts a 64-byte cache line):

Offset Type Field Owner
384 u64 atomic writeIndex producer
392 u32 atomic blockFrames producer
448 u64 atomic readIndex consumer
512 u64 atomic overrunCount diagnostics
576 u64 atomic underrunCount diagnostics
584 u32 atomic consumerPid consumer, optional
588 char[32] consumerName consumer, optional
624 u64 consumerStart consumer process start stamp, 0 = unknown (§7)

blockFrames is the producer's last block size in frames, published with a relaxed store before the writeIndex release that makes it visible. It shares the producer's cache line, which the producer already writes every block. 0 means unknown: the producer predates the field, or has not pushed yet. Consumers size their prefill cushion from it and must treat any value above the header's maxBlockSize as unknown too. A producer that predates the field zero-initialises the whole region, so a stale value cannot survive a re-init.

Indices are monotonic frame counts (never wrapped); position in the ring is index & (capacity-1). 64-bit indices cannot realistically overflow (6.5M years @ 96 kHz).

Producer push (per audio block, ≤ maxBlockSize ≤ capacity frames):

  1. w = writeIndex.load(relaxed)
  2. per channel: copy frames to audioData[c*capacity + (w & mask)], splitting at the wrap boundary (two memcpys)
  3. writeIndex.store(w + frames, release)

The producer never reads readIndex and never blocks: if the consumer is slow, it simply keeps writing (the consumer detects the loss, see below).

Consumer pop:

  1. r = readIndex.load(relaxed); w = writeIndex.load(acquire)
  2. available = w - r. If available > capacity, data was overwritten: set r += available - capacity, available = capacity, increment overrunCount (relaxed). This is drop-oldest recovery: the consumer skips to the oldest still-valid frame.
  3. copy min(available, wanted) frames out (planar → whatever you need), splitting at the wrap boundary
  4. readIndex.store(r + copied, release)

underrunCount is a consumer-side diagnostic (incremented when it needed frames and none were available); the reference pop does not touch it.

SPSC contract: exactly one consumer per stream. readIndex has a single owner; a second simultaneous consumer corrupts both. On first open, a consumer should fast-forward (readIndex = writeIndex) to skip stale audio.

Consumer claim (optional). A consumer MAY write its process id to consumerPid on open (name and consumerStart first, then the pid with a release store) and clear it on close, clearing only if the value is still its own. 0 means unclaimed. It exists because a frozen readIndex is ambiguous: a consumer that is attached but not pulling right now looks exactly like one that left. A producer that cares (Aether's plugin shows a reader-present indicator) reads the claim, checks the pid is live, and SHOULD also check that consumerStart still matches that process (§7), treating a stale claim as no claim. Aether's plugin and CHOP do both. A consumer clears consumerName and consumerStart when it releases the slot, so neither outlives the claim it belonged to. Producers MUST NOT decide anything from consumerName: it is display only, and a consumer that leaves one behind is still a valid consumer. Producers MUST NOT require the claim itself either: a consumer that never writes one is still valid, and the producer falls back to watching readIndex move. The producer zeroes the whole region when it re-initialises the segment, so a consumer must re-claim after a re-init (it re-opens anyway, see §8). A consumer that claims SHOULD refuse to open a ring that already carries a live claim: its fast-forward drains the holder's cushion, and its later release clears the holder's claim while the holder is still attached. A long-lived consumer SHOULD re-check its claim while connected and re-write it if it was cleared, since a consumer that ignores the claim can still do this. Aether's CHOP does both; its Python reader refuses.

6. Discovery registry

Optional but recommended. One well-known segment (ShmAudioRegistry, 64 + 16×192 = 3136 bytes) listing active streams.

Registry header (64 bytes): magic u32 = 0xA0D1FE60, version u32 = 1, maxEntries u32 = 16, activeCount u32, reserved[48].

Entry (192 bytes each, first at offset 64):

Offset Type Field
0 char[64] streamName
64 char[32] producerName
96 u32 producerPid
100 u32 numChannels
104 f64 sampleRate
112 u32 ringCapacity
116 u32 active (1 = in use)
120 u64 heartbeat (opaque timestamp, refreshed ~1/s)
128 u64 ownerStart
136 u8[56] reserved

Mutations (register / heartbeat / deregister) are read-modify-write sequences and must hold the cross-process registry lock (§1). Enumeration may read without the lock. Neither name is guaranteed NUL-terminated: a reader must terminate its own copy before treating either as a C string.

A slot index is not proof of ownership. Registration may reclaim the slot of a producer that died without deregistering, so a process holding an index from earlier can be pointing at somebody else's entry. Before heartbeating or deregistering, match both producerPid and ownerStart against what you stamped at registration; 0 on either side means unknown, and pid alone decides there. Producers: register after creating the stream, heartbeat ~1/s, deregister on clean shutdown. Consumers: treat an entry whose producerPid is dead as stale (crashed producers leave entries behind).

7. Lifecycle & liveness

Producer: create segment → init header/transport/ring meta → zero audio → (Windows) create+signal ready event → register in registry. On shutdown: deregister → set header magic = 0 (invalidates the segment for consumers that still hold a mapping) → (POSIX) shm_unlink → unmap.

Consumer: try open; if absent, wait. Windows: wait on the ready event (create it unsignaled if it doesn't exist yet, the producer's create signals it); POSIX: poll shm_open at ~50 ms. Validate header (§3). Fast-forward readIndex. Poll writeIndex for new frames (typical cadence 1 ms or host-block-rate).

Staleness (re-configuration): a producer may destroy and re-create a stream in place, with a different channel count or capacity, while its process keeps running (so producerPid liveness does not catch this). Cache createdTimestamp at open and treat the segment as ended when the header becomes invalid (magic zeroed) or createdTimestamp no longer matches; then close and re-open, re-reading numChannels/ringCapacity. This matters especially on Windows, where a consumer's open handle pins the named section alive: the producer's re-create maps and re-initializes the OLD section in place, so segment identity alone never signals the change. Reference: ShmAudioReader::isStale().

Liveness: hostTimestamp/heartbeat values are platform-clock-specific (Windows QPC ticks, macOS mach_absolute_time, Linux CLOCK_MONOTONIC ns), so treat them as opaque change-detectors, not wall-clock. Cross-process liveness uses producerPid: OpenProcess+GetExitCodeProcess (Windows) or kill(pid, 0) (POSIX).

A pid alone is not enough. The OS reuses pids, so a crashed peer's pid can come back as an unrelated live process and its stale stamp then reads as a holder forever (the stream name stays refused, the ring stays claimed). Writers therefore stamp a start token beside the pid: producerStartToken in the header, consumerStart in the ring meta. It is the process creation time, opaque and platform-specific (Windows GetProcessTimes FILETIME, macOS sysctl KERN_PROC_PID p_starttime µs, Linux /proc/<pid>/stat field 22); only equality is ever compared, never its magnitude, and it is never compared across machines. A holder is live when the pid is live and the token read back from that pid equals the stamped one.

Both fields are additive, no version bump: 0 means the writer predates them or its platform supplies no start time, and a reader MUST then fall back to pid-only rather than treating the mismatch as dead. A reader that cannot query the token now falls back the same way.

Segment lifetime. Windows: the mapping dies with its last open handle; a crashed producer's segment vanishes once consumers close. POSIX: named shm persists until shm_unlink; the producer unlinks on clean close, but a crash leaks the name (next shmCreate reuses/re-truncates it).

Not defended in v1. The trust boundary is the local machine and the user's own session: every check in this spec is about truncated, stale or version-skewed segments, not about a hostile local writer. The namespace is open, so anything running as the same user can take a stream name before the producer does (squatting), write a consumerPid and consumerStart it does not own (claim forging), zero a header's magic to invalidate a live segment, or on POSIX ftruncate a named object other processes have mapped. None of these is closed by a field check, and the validation in §3 does not attempt it: a forged claim is well-formed by construction. They are availability risks, not confidentiality ones, and a v2 that wanted to close them would need OS-level access control on the names, not more fields.

8. Limits (v1)

Constant Value
kMaxChannels 16
kMinRingCapacity / kMax 2048 / 65536 frames
kMaxSampleRate 768000 Hz
maxBlockSize ≤ 65536 frames, 0 = unknown
kMaxStreamName / kMaxProducerName 64 / 32 bytes (incl. NUL)
kMaxRegistryEntries 16
sampleFormat float32 only

Verified envelope: {1,2,4,8,16} channels × {44.1, 48, 88.2, 96 kHz} bit-exact-continuous under consumer/run_matrix.ps1.

9. Versioning

protocolVersion bumps only on breaking layout changes; reserved fields may gain meaning in minor revisions (writers must zero them, readers must not require them). Reject segments whose magic or version you don't recognize.