Lock-free, single-producer / single-consumer (SPSC) protocol for streaming float32 audio + DAW transport between processes over OS shared memory. No daemon, no sockets, no kernel round-trips on the data path.
License: MIT. This specification may be freely implemented, copied, and redistributed. MIT-licensed reference implementations (C++ header-only and Python) live alongside this file in the Aether source tree and are available from the author.
- Byte order: little-endian. Target archs: x86-64, ARM64.
- All offsets are from the start of the shared memory segment.
u32/u64/i32/f32/f64= fixed-width little-endian scalars.- Atomic fields require lock-free 64-bit atomics (C11
stdatomic, C++std::atomic, orInterlocked*).
1. Naming
| Object | Windows | POSIX (macOS/Linux) |
|---|---|---|
| Audio segment | ShmAudio_<stream> (CreateFileMapping) |
/ShmAudio_<stream> (shm_open) |
| Ready event | ShmAudio_<stream>_Ready (manual-reset event) |
none, poll shm_open |
| Registry segment | ShmAudioRegistry |
/ShmAudioRegistry |
| Registry lock | Local\ShmAudioRegistryLock (mutex) |
/ShmAudioRegistryLk (sem_open, init 1) |
<stream> = user-chosen stream name, ≤63 bytes, null-terminated in headers.
POSIX name clamp: macOS limits shm names to 31 chars (PSHMNAMLEN). Names
are truncated deterministically to 31 chars after prefixing, so producer and
consumer must apply the same truncation. Consequence: stream names sharing a
21-char prefix collide on macOS.
2. Audio segment layout
Total size = 4096 + ringCapacity × numChannels × 4 bytes.
| Offset | Size | Region |
|---|---|---|
| 0 | 256 | Header (§3), written once by producer |
| 256 | 128 | Transport state (§4), seqlock |
| 384 | 256 | Ring metadata (§5), atomics, one per cache line |
| 640 | 3456 | reserved (zero) |
| 4096 | capacity×channels×4 | Audio data (§5), planar float32, page-aligned |
3. Header (256 bytes, offset 0)
Written by the producer before the segment name/ready event becomes visible; read-only for consumers.
| Offset | Type | Field | Notes |
|---|---|---|---|
| 0 | u32 | magic | 0xA0D10500 |
| 4 | u32 | protocolVersion | 1 |
| 8 | u32 | headerSize | 256 |
| 12 | u32 | totalSegmentSize | bytes |
| 16 | f64 | sampleRate | e.g. 48000.0 |
| 24 | u64 | createdTimestamp | opaque, platform clock (§7) |
| 32 | u32 | maxBlockSize | max frames per producer push |
| 36 | u32 | numChannels | 1–16 |
| 40 | u32 | sampleFormat | 0 = float32 (only format in v1) |
| 44 | u32 | ringCapacity | frames, power of two, 2048–65536 |
| 48 | u32 | audioDataOffset | 4096 |
| 52 | u32 | transportOffset | 256 |
| 56 | u32 | ringMetaOffset | 384 |
| 60 | u32 | producerPid | for liveness (§7) |
| 64 | char[64] | streamName | null-terminated |
| 128 | char[32] | producerName | null-terminated, informational |
| 160 | u64 | producerStartToken | producer process start stamp, 0 = unknown (§7) |
| 168 | u8[88] | reserved | zero |
Validation on open (reject segment otherwise): magic, version, and
headerSize match; ringCapacity is a power of two within §8's limits;
0 < numChannels ≤ 16; 0 < sampleRate ≤ 768000; maxBlockSize ≤ 65536
(0 is legal and means unknown); the three offset fields equal the §2
constants; totalSegmentSize == 4096 + capacity×channels×4. Then check the
size of the object you actually mapped (VirtualQuery / fstat) and reject
anything shorter than totalSegmentSize: a header that outruns its segment is
truncated or hostile, and every pointer below is derived from the §2 constants
these fields must match. v1 fixes the layout at those constants; moving it is a
version bump, so a reader may hardcode them and validate the fields against
them, which is what the reference readers do.
4. Transport state (128 bytes, seqlock)
Producer publishes DAW transport once per audio block. Guarded by a seqlock:
odd sequence = write in progress.
| Offset | Type | Field |
|---|---|---|
| 0 | u32 atomic | sequence |
| 4 | u32 | pad |
| 8 | f64 | bpm |
| 16 | f64 | samplePos (samples from project start) |
| 24 | f64 | ppqPos (quarter notes) |
| 32 | f64 | lastBarPpq |
| 40 | i32 | timeSigNumerator |
| 44 | i32 | timeSigDenominator |
| 48 | u32 | transportFlags: bit0 playing, bit1 recording (reserved: the Aether producer never sets it), bit2 looping |
| 52 | u32 | pad |
| 56 | u64 | hostTimestamp (opaque, platform clock) |
| 64 | u8[64] | reserved |
Writer (producer only):
sequence.store(seq+1, relaxed);atomic_thread_fence(release)- write payload
sequence.store(seq+2, release)
The release fence after the odd store is required. A plain release store only orders writes that precede it, not the payload writes that follow.
Reader:
s1 = sequence.load(acquire); if odd, retry- copy payload
atomic_thread_fence(acquire);s2 = sequence.load(acquire)s1 == s2→ snapshot is consistent, else retry (bounded, e.g. 64 attempts)
5. Audio ring buffer
Data layout is planar: channel c occupies
audioData[c*capacity .. c*capacity + capacity-1] (float32 frames). This lets
both sides use straight memcpy per channel.
Ring metadata (offset 384; each OWNER starts a 64-byte cache line):
| Offset | Type | Field | Owner |
|---|---|---|---|
| 384 | u64 atomic | writeIndex | producer |
| 392 | u32 atomic | blockFrames | producer |
| 448 | u64 atomic | readIndex | consumer |
| 512 | u64 atomic | overrunCount | diagnostics |
| 576 | u64 atomic | underrunCount | diagnostics |
| 584 | u32 atomic | consumerPid | consumer, optional |
| 588 | char[32] | consumerName | consumer, optional |
| 624 | u64 | consumerStart | consumer process start stamp, 0 = unknown (§7) |
blockFrames is the producer's last block size in frames, published with a
relaxed store before the writeIndex release that makes it visible. It shares
the producer's cache line, which the producer already writes every block. 0
means unknown: the producer predates the field, or has not pushed yet.
Consumers size their prefill cushion from it and must treat any value above the
header's maxBlockSize as unknown too. A producer that predates the field
zero-initialises the whole region, so a stale value cannot survive a re-init.
Indices are monotonic frame counts (never wrapped); position in the ring is
index & (capacity-1). 64-bit indices cannot realistically overflow
(6.5M years @ 96 kHz).
Producer push (per audio block, ≤ maxBlockSize ≤ capacity frames):
w = writeIndex.load(relaxed)- per channel: copy frames to
audioData[c*capacity + (w & mask)], splitting at the wrap boundary (two memcpys) writeIndex.store(w + frames, release)
The producer never reads readIndex and never blocks: if the consumer is
slow, it simply keeps writing (the consumer detects the loss, see below).
Consumer pop:
r = readIndex.load(relaxed);w = writeIndex.load(acquire)available = w - r. Ifavailable > capacity, data was overwritten: setr += available - capacity,available = capacity, incrementoverrunCount(relaxed). This is drop-oldest recovery: the consumer skips to the oldest still-valid frame.- copy
min(available, wanted)frames out (planar → whatever you need), splitting at the wrap boundary readIndex.store(r + copied, release)
underrunCount is a consumer-side diagnostic (incremented when it needed
frames and none were available); the reference pop does not touch it.
SPSC contract: exactly one consumer per stream. readIndex has a single
owner; a second simultaneous consumer corrupts both. On first open, a consumer
should fast-forward (readIndex = writeIndex) to skip stale audio.
Consumer claim (optional). A consumer MAY write its process id to
consumerPid on open (name and consumerStart first, then the pid with a
release store) and clear it on close, clearing only if the value is still its
own. 0 means unclaimed.
It exists because a frozen readIndex is ambiguous: a consumer that is attached
but not pulling right now looks exactly like one that left. A producer that
cares (Aether's plugin shows a reader-present indicator) reads the claim,
checks the pid is live, and SHOULD also check that consumerStart still matches
that process (§7), treating a stale claim as no claim. Aether's plugin and CHOP
do both. A consumer clears consumerName and consumerStart when it releases
the slot, so neither outlives the claim it belonged to. Producers MUST NOT
decide anything from consumerName: it is display only, and a consumer that
leaves one behind is still a valid consumer. Producers MUST NOT require the
claim itself either: a consumer that never writes one is still valid, and the
producer falls back to watching readIndex move. The producer zeroes the whole
region when it re-initialises the segment, so a consumer must re-claim after a
re-init (it re-opens anyway, see §8).
A consumer that claims SHOULD refuse to open a ring that already carries a live
claim: its fast-forward drains the holder's cushion, and its later release
clears the holder's claim while the holder is still attached. A long-lived
consumer SHOULD re-check its claim while connected and re-write it if it was
cleared, since a consumer that ignores the claim can still do this. Aether's
CHOP does both; its Python reader refuses.
6. Discovery registry
Optional but recommended. One well-known segment (ShmAudioRegistry,
64 + 16×192 = 3136 bytes) listing active streams.
Registry header (64 bytes): magic u32 = 0xA0D1FE60, version u32 = 1,
maxEntries u32 = 16, activeCount u32, reserved[48].
Entry (192 bytes each, first at offset 64):
| Offset | Type | Field |
|---|---|---|
| 0 | char[64] | streamName |
| 64 | char[32] | producerName |
| 96 | u32 | producerPid |
| 100 | u32 | numChannels |
| 104 | f64 | sampleRate |
| 112 | u32 | ringCapacity |
| 116 | u32 | active (1 = in use) |
| 120 | u64 | heartbeat (opaque timestamp, refreshed ~1/s) |
| 128 | u64 | ownerStart |
| 136 | u8[56] | reserved |
Mutations (register / heartbeat / deregister) are read-modify-write sequences and must hold the cross-process registry lock (§1). Enumeration may read without the lock. Neither name is guaranteed NUL-terminated: a reader must terminate its own copy before treating either as a C string.
A slot index is not proof of ownership. Registration may reclaim the slot of a
producer that died without deregistering, so a process holding an index from
earlier can be pointing at somebody else's entry. Before heartbeating or
deregistering, match both producerPid and ownerStart against what you
stamped at registration; 0 on either side means unknown, and pid alone
decides there. Producers: register after creating the stream, heartbeat
~1/s, deregister on clean shutdown. Consumers: treat an entry whose
producerPid is dead as stale (crashed producers leave entries behind).
7. Lifecycle & liveness
Producer: create segment → init header/transport/ring meta → zero audio →
(Windows) create+signal ready event → register in registry. On shutdown:
deregister → set header magic = 0 (invalidates the segment for consumers
that still hold a mapping) → (POSIX) shm_unlink → unmap.
Consumer: try open; if absent, wait. Windows: wait on the ready event
(create it unsignaled if it doesn't exist yet, the producer's create signals
it); POSIX: poll shm_open at ~50 ms. Validate header (§3). Fast-forward
readIndex. Poll writeIndex for new frames (typical cadence 1 ms or
host-block-rate).
Staleness (re-configuration): a producer may destroy and re-create a
stream in place, with a different channel count or capacity, while its
process keeps running (so producerPid liveness does not catch this). Cache
createdTimestamp at open and treat the segment as ended when the header
becomes invalid (magic zeroed) or createdTimestamp no longer matches; then
close and re-open, re-reading numChannels/ringCapacity. This matters
especially on Windows, where a consumer's open handle pins the named section
alive: the producer's re-create maps and re-initializes the OLD section in
place, so segment identity alone never signals the change. Reference:
ShmAudioReader::isStale().
Liveness: hostTimestamp/heartbeat values are platform-clock-specific
(Windows QPC ticks, macOS mach_absolute_time, Linux CLOCK_MONOTONIC ns), so
treat them as opaque change-detectors, not wall-clock. Cross-process liveness
uses producerPid: OpenProcess+GetExitCodeProcess (Windows) or
kill(pid, 0) (POSIX).
A pid alone is not enough. The OS reuses pids, so a crashed peer's pid can come
back as an unrelated live process and its stale stamp then reads as a holder
forever (the stream name stays refused, the ring stays claimed). Writers
therefore stamp a start token beside the pid: producerStartToken in the
header, consumerStart in the ring meta. It is the process creation time,
opaque and platform-specific (Windows GetProcessTimes FILETIME, macOS
sysctl KERN_PROC_PID p_starttime µs, Linux /proc/<pid>/stat field 22);
only equality is ever compared, never its magnitude, and it is never compared
across machines. A holder is live when the pid is live and the token read
back from that pid equals the stamped one.
Both fields are additive, no version bump: 0 means the writer predates them
or its platform supplies no start time, and a reader MUST then fall back to
pid-only rather than treating the mismatch as dead. A reader that cannot query
the token now falls back the same way.
Segment lifetime. Windows: the mapping dies with its last open handle; a
crashed producer's segment vanishes once consumers close. POSIX: named shm
persists until shm_unlink; the producer unlinks on clean close, but a crash
leaks the name (next shmCreate reuses/re-truncates it).
Not defended in v1. The trust boundary is the local machine and the user's
own session: every check in this spec is about truncated, stale or
version-skewed segments, not about a hostile local writer. The namespace is
open, so anything running as the same user can take a stream name before the
producer does (squatting), write a consumerPid and consumerStart it does not
own (claim forging), zero a header's magic to invalidate a live segment, or on
POSIX ftruncate a named object other processes have mapped. None of these is
closed by a field check, and the validation in §3 does not attempt it: a forged
claim is well-formed by construction. They are availability risks, not
confidentiality ones, and a v2 that wanted to close them would need OS-level
access control on the names, not more fields.
8. Limits (v1)
| Constant | Value |
|---|---|
| kMaxChannels | 16 |
| kMinRingCapacity / kMax | 2048 / 65536 frames |
| kMaxSampleRate | 768000 Hz |
| maxBlockSize | ≤ 65536 frames, 0 = unknown |
| kMaxStreamName / kMaxProducerName | 64 / 32 bytes (incl. NUL) |
| kMaxRegistryEntries | 16 |
| sampleFormat | float32 only |
Verified envelope: {1,2,4,8,16} channels × {44.1, 48, 88.2, 96 kHz}
bit-exact-continuous under consumer/run_matrix.ps1.
9. Versioning
protocolVersion bumps only on breaking layout changes; reserved fields may
gain meaning in minor revisions (writers must zero them, readers must not
require them). Reject segments whose magic or version you don't recognize.