Audio Online — Step 1: engine and dependency report

Status: awaiting Curt's sign-off. Nothing has been vendored and no app code exists yet. Date: 2026-10-30. Spike harness and raw results: spikes/step1/.

---

0. Requirements no candidate can meet (read first)

1. The sandbox cannot run Web Audio. HashBin SDK v0.4.0's createSandbox() runs code only in a

classic worker inside the opaque iframe. The iframe window runs only the SDK's fixed bootstrap. OfflineAudioContext, AudioContext and AudioWorklet are undefined in that worker in all three engines (spike, §2.1). "Mix in the sandbox iframe's window" would need a new SDK mode or an app-owned frame bootstrap (Livelier does this). Proposed resolution: do all rendering in the worker with our own deterministic DSP. Native Web Audio is then used only for playback on the editor origin. This also gives better determinism (item 3).

1. WebCodecs is not available in the sandbox worker on Chromium. Chromium reports

isSecureContext: false for the SDK's blob worker in an opaque frame, so AudioEncoder and crypto.subtle are absent. Firefox and WebKit expose them. Proposed resolution: the sandbox returns PCM, and compressed encoding runs on the editor origin. That code is ours and only processes bounded Float32Array data, never user code.

1. Native Web Audio is not bit-exact across engines. Every node except a plain buffer copy gives

different bytes in Chromium, Firefox and WebKit. Firefox's resampler is materially different: the peak of resampled white noise is 1.31 there versus 1.94 elsewhere (§2.2). Within one engine it is repeatable. Rendering with native nodes would therefore meet the brief's per-engine rule, but never a cross-engine one.

1. No permissive neural voice stack exists off the shelf. Every maintained browser wrapper for

Kokoro (kokoro-js), Kitten TTS (kitten-tts-js) and Piper (piper-tts-web and its forks) converts text to phonemes with eSpeak NG, which is GPL-3.0. phonemizer@1.2.1 is labelled Apache-2.0 on npm, but its 1.3 MB bundle is a compiled eSpeak NG. Upstream Piper's successor (OHF-Voice/piper1-gpl) is GPL-3.0 as well. The model weights themselves are permissive. Curt must choose one of three options: accept a GPL phonemizer that is downloaded on demand, have us write an original CMUdict-based G2P, or skip neural voices in v1 (§4.3, decision D3).

1. eSpeak NG cannot be the bundled voice under MIT-only terms. It is GPL-3.0. The only small,

permissive bundled engine with acceptable quality is SVOX Pico (Apache-2.0). It supports 6 languages instead of 100+. Debian classes Pico's language files as non-free (§4.1).

1. Neural-voice determinism is unverified. VITS (Piper) and Kokoro's vocoder contain random-noise

ops. Bit-exact output needs either seeded or zeroed noise inputs (to verify in Step 3) or a verified render cache keyed by (text, voice, model hash). I can't promise it yet.

1. Not verified on real Safari. WebKit results come from Playwright WebKit 27.2. Driving Safari

would need *Allow Remote Automation*, a system setting I must not change. Historically Safari's AudioEncoder support lagged, so WAV export is the guaranteed fallback.

1. Strudel compatibility is not on offer. Strudel is AGPL-3.0, so the pattern language must be an

original implementation of a Tidal-style mini-notation subset. It won't be Strudel-compatible.

---

1. Recommendation in brief

• One sandbox worker renders everything. Python (Pyodide, only when needed) or a notation parser

produces a timeline JSON graph. An original JS renderer in the same worker realizes it with deterministic DSP. The worker returns bounded PCM (Float32Array per channel, with declared length and sample rate).

• Deterministic DSP: only IEEE-exact operations (+ − × ÷ sqrt floor, Math.fround, Math.pow

and Math.sqrt matched too) plus our own dmath sin/cos/exp/log/tanh. Third-party JS synths (ZzFX, jsfxr) load unmodified, with a module-scope Math shadow prepended at load time, so the vendored files stay byte-identical. WASM engines (Pyodide/numpy, Pico) are already bit-exact across engines. Result: identical bytes on Chromium, Firefox and WebKit, which is stronger than the brief asks. The spike proved this for every engine-side piece we would ship (§2.2).

• Editor origin: playback (AudioContext, started from a gesture), waveform/spectrogram, WebCodecs

encoding, and muxing with original code extended from ManimOnline's manim-video.js.

• Pure-notation documents never load Pyodide. Parsers are about 50 KB to 0.5 MB and the first

render takes milliseconds. Python documents add about 16.4 MB of downloads and about 1.2 s of warm startup.

• Bundled voice: SVOX Pico (Apache-2.0, 2.5 MB for en-US, bit-exact, ~30× real time). **Neural

voices on demand:** Kokoro-82M (86 MB, Apache-2.0) and/or Kitten TTS nano (24 MB, Apache-2.0) on onnxruntime-web (MIT, 14.8 MB WASM). The phonemizer depends on decision D3.

---

2. Spikes (run 2026-10-30 on macOS 27.0.1, Apple silicon)

Engines: Chromium 156.0.8078.4 (Playwright 1.64.1), Firefox 159.0 (desktop build through WebDriver BiDi, because Playwright's Firefox Nightly 157 fails at launch on this Mac with "Could not find profile folder"; hashbin.org's owned-app doc reports the same failure), and WebKit 27.2 (Playwright). The page used the real vendored SDK v0.4.0, a sibling-style CSP (SDK bootstrap hash, blob:, 'wasm-unsafe-eval') and real engines throughout, with no mocks. Reproduce with:

spikes/step1/fetch-assets.sh
PW=../GnuPlotOnline/node_modules/playwright node spikes/step1/run.cjs chromium webkit firefox-bidi
node spikes/step1/compare.cjs

It ran three contexts on each engine: (A) the editor origin, (B) a raw sandbox="allow-scripts" srcdoc iframe window with its own hashed bootstrap (Livelier-style; not SDK), and (C) the SDK's opaque worker.

2.1 Web Audio and platform APIs per context

Conclusions: no Web Audio is available in the sandbox. A window-mode sandbox would get OfflineAudioContext, but not worklets on WebKit. WebKit's opaque worker can reach a caches object. That is its own opaque partition, not the editor's, but the app should still delete storage APIs before running user code, as PythonOnline's harden() does. The browser isolation test will check this.

2.2 Determinism

Each case rendered twice per engine. Every case repeated byte-for-byte within each engine. The question is whether bytes match *across* engines:

Interpretation: anything that uses only basic float operations or WASM gives identical bytes on all three engines. Anything that touches an engine's libm or native DSP does not.

2.3 Encoders (AudioEncoder.isConfigSupported, editor origin)

All encoders gave identical bytes on repeat. Opus in WebM is the only lossy format every engine can write. AAC/M4A is available on Chromium and WebKit.

2.4 Speed and startup (warm local server, inside the SDK worker)

Everything in the worker renders far faster than real time. Pyodide's cost is its downloads: pyodide.asm.wasm 9,598,218 + python_stdlib.zip 2,545,637 + pyodide.asm.mjs 1,250,344 + numpy wheel 2,960,568 bytes ≈ 16.4 MB, plus about 1.2 s of startup. Notation-only documents avoid both.

---

3. Candidate matrix

Sizes are raw bytes (gzip where measured). "Worker" means it runs in the SDK's opaque classic worker. Release dates are for the named version.

3.1 Timeline / mixer

3.2 Music

3.3 SFX

3.4 Voice

3.5 Encoding / containers

Fallback when AudioEncoder is missing: WAV export stays available, and compressed export is disabled with an explanation. Gallery previews use <audio> with two <source> elements, WebM/Opus first and M4A/AAC second, both generated by render-gallery.mjs on macOS Chromium.

---

4. Voice size and licensing detail

4.1 Bundled: SVOX Pico

• Engine source is Apache-2.0 (SVOX AG / AOSP). The language files (pico/lang/*.bin) are

distributed by AOSP as Apache-2.0, but Debian packages them as *non-free*: "Claims to be Apache-2.0 but the source code is not included. At least, it is redistributable." There is no restriction on redistribution or commercial use, only a missing "preferred form for modification". Curt should accept this explicitly (D2).

• en-US totals 2,543,129 bytes (about 1.5 MB gzip est.). en-GB, de-DE, es-ES, fr-FR and it-IT each

add 0.86–1.2 MB. Output is 16 kHz mono 16-bit. The spike utterance hit full scale (peak 32768), so the renderer applies headroom and a deterministic 16→48 kHz resampler.

• Quality is better than eSpeak and clearly synthetic.

4.2 Neural (on demand; not part of "Available offline")

The editor origin downloads these after confirmation, verifies them against pinned SHA-256 hashes, and stores them in a separate cache (audio-voices-v1). They reach the sandbox through the SDK proxy, as Pyodide does. kokoro.web.js loads ORT WASM from jsDelivr by default, so wasmPaths must point at our verified copy.

4.3 The G2P decision (D3)

• (a) GPL phonemizer on demand. Load phonemizer@1.2.1 (compiled eSpeak NG, effectively

GPL-3.0) as a separate, unmodified, downloaded module with its own notice. Whether this is "mere aggregation" under GPL is a legal judgment for Curt. Fastest path.

• (b) Original permissive G2P. CMUdict (BSD-style, ~3.6 MB raw / ~0.8 MB gzip) mapped

ARPAbet→IPA, plus original letter-to-sound fallback rules. Fits the house license and adds about 2–3 days of work. Pronunciation may differ slightly from what Kokoro/Kitten were trained on.

• (c) No neural voices in v1.

My recommendation is (b), using (a) temporarily only if Curt is comfortable with it.

---

5. Test tolerances (follow from §2.2)

• Rendered PCM from the worker: exact. SHA-256 of the rendered float32 PCM (and WAV export) must

match a committed golden hash on all three engines, not only within one. This applies to every gallery example except neural voices. The diagnostic checks below still run, so a mismatch is easy to read: - duration: exact sample count; - channels: exact; - silence boundaries: within ±10 ms at −60 dBFS; - peak/RMS per 100 ms window: ±0.1 dB.

• Determinism test: render twice in one session and require byte equality.
• Compressed exports: within-engine repeat equality only. Cross-engine checks decode the file and

compare duration to within one codec frame plus priming (Opus pre-skip 312; AAC 1024/2112) and RMS to within 1 dB.

• Neural voices: per-engine byte equality if Step 3 shows seeded or zeroed noise works.

Otherwise, cache equality keyed by (text, voice, model hash), and tests assert duration ±5% and non-silence only.

• Native Web Audio: not used for rendering, so there are no tolerances to set.

---

6. (a) Recommended dependency list

Bundled and offline-cached. Pinned and hash-verified. Where siblings do so, it is fetched from the CDN rather than committed (see §8).

Original code (MIT): audio-lite.py, the timeline schema, the renderer/DSP, dmath, the resampler, the WAV reader/writer, the muxers (extending ManimOnline), the pattern mini-notation and built-in drum kit, ABC instruments, the detector and the gallery.

On demand, after confirmation (not offline):

Rejected: Tone.js, Strudel, WebChucK, native-Web-Audio rendering, eSpeak NG as the bundled voice (unless D2 says otherwise), SAM, Flite WASI, mp4-muxer/webm-muxer, mediabunny and lamejs. Csound, Faust and music21 are deferred.

Decisions for Curt

• D1. Architecture: render everything in the sandbox worker with original deterministic DSP, and

encode on the editor origin.

• D2. Bundled voice: Pico (recommended; accept the lingware caveat), or eSpeak NG (GPL), or both

(Pico default, eSpeak as an opt-in GPL download).

• D3. Neural G2P: (b) original CMUdict (recommended), (a) GPL phonemizer, or (c) none in v1.
• D4. Neural models: Kitten nano (smaller) and Kokoro (better), or only one of them.
• D5. Uploaded compressed audio (MP3/OGG/M4A). Decoding it needs the editor origin's

decodeAudioData, which differs per engine. Recommendation: accept WAV only (deterministic) in v1, decoded in the sandbox. Larger audio goes through CID samples.

• D6. Pyodide served like PythonOnline (pinned jsDelivr URL plus verified offline cache) rather

than committed to the repo, as the brief literally says.

• D7. The HashBin app ID: placeholder app_00000000-0000-0000-0000-000000000000 until supplied.

---

7. (c) audio-lite API proposal

Split: Python *describes* and JS *renders*. construct() builds a plain-JSON timeline. Raw numpy arrays travel as transferred Float32Array buffers referenced by id. The JS renderer realizes the timeline in the same worker. Notation-only documents build the same timeline directly in JS.

from audio import *

class Intro(Scene):
    sample_rate = 48000            # 44100 | 48000
    channels = 2
    seed = 1                       # all randomness derives from this
    # length = 12                  # optional; otherwise the last clip end plus effect tails

    def construct(self):
        self.say("Welcome to Audio Online.", voice="pico:en-US", track="voice")   # advances self.time
        theme = abc("X:1\nT:Theme\nK:C\nCEG c2|", instrument="pluck")
        self.add(theme, at=0.5, track="music", gain=-12).fade_in(1)
        boom = Sine(55).env(adsr(0.01, 0.2, 0, 0.3)).dur(0.6) + Noise(seed=1).lowpass(800).dur(0.3)
        self.add(boom, at=self.time, track="sfx", pan=-0.3)
        self.wait(1)
        self.track("music").fx(Reverb(room=0.4), Compressor(-18, 4)).duck(under="voice", amount=-9)
        self.master.fx(Limiter(-1))

• Sources (nouns):

- oscillators: Sine/Square/Saw/Triangle(freq); - noise: Noise(color="white|pink|brown", seed=); - envelopes: adsr(a, d, s, r), Envelope([(t, v), …], curve=); - samples: Samples(array, rate), Sample(cid="…"), Sample("kick.wav") (uploaded file); - notation: abc(text, instrument=), zzfx([...]), sfxr(params | preset, seed=), pattern("bd*2 [~ sn] hh*4", bpm=, kit=); - speech: Speech(text, voice=); - Silence(seconds).

• Clip operations return new clips, immutable as in Pydub:

- .dur(s), [a:b] (seconds, unlike Pydub's ms), .loop(n | until=), .gain(db), .pan(p); - .fade_in(s), .fade_out(s), .reverse(), .speed(r), .env(e); - .lowpass/.highpass/.bandpass(f, q), .fx(*effects); - a + b mixes (as in the brief's sketch, not Pydub's concatenation), a >> b sequences, a * n repeats.

• Placement and time (Manim verbs):

- self.add(clip, at=self.time, track="main", gain=0, pan=0) returns a placement handle; - self.play(clip) adds the clip and advances self.time by its length; - self.wait(s), self.say(text, voice=, …) (equivalent to play(Speech(...))), self.time.

• Tracks and master:

- self.track(name) supports .fx(...), .gain(db), .pan(p), .duck(under=, amount=, attack=, release=) and automation, e.g. .automate("gain", [(0, -60), (2, 0)]); - self.master takes the same effects.

• Effects: LowPass/HighPass/BandPass/Peak/Shelf, Delay(time, feedback, mix),

Reverb(room, damp, mix) (Freeverb-style, our implementation), Compressor(threshold, ratio, attack, release, makeup), Limiter(ceiling), Gain, Drive. All of them are deterministic JS.

• Speech length inside construct(): say() must know the utterance length synchronously.

Pico is synchronous WASM in the same worker, so a bridge synthesizes and caches during construct(). Neural voices are async, so construct() runs in a collect pass that records speech requests with estimated lengths. The app then synthesizes them and re-runs construct() (it is deterministic because seeded) until the lengths are stable, at most 3 passes. Users write ordinary synchronous code either way.

• Source lines: every clip records its caller's line (sys._getframe) so lane clicks jump to

source. Tracebacks use the user file name with no prepended lines, as Gnuplot's diagnostics do.

• Several Scenes in one file show a scene selector, as in ManimOnline.

Timeline JSON (abbreviated):

{"format":"audio-timeline","version":1,"sampleRate":48000,"channels":2,"seed":1,"length":null,
 "sources":{"s1":{"kind":"osc","wave":"sine","freq":55,"dur":0.6,"env":{"adsr":[0.01,0.2,0,0.3]},"line":9},
            "s2":{"kind":"speech","text":"Welcome to Audio Online.","voice":"pico:en-US","line":6},
            "s3":{"kind":"buffer","ref":0,"rate":22050}},
 "clips":[{"id":"c1","source":"s1","track":"sfx","at":1.82,"gain":0,"pan":-0.3,"fade":[0,0],"slice":[0,null],"loop":1,"fx":[],"line":10}],
 "tracks":{"music":{"fx":[{"type":"reverb","room":0.4},{"type":"compressor","threshold":-18,"ratio":4}],"duck":{"under":"voice","amount":-9}}},
 "master":{"fx":[{"type":"limiter","ceiling":-1}]}}

Saved documents follow the brief: { format: 'hashbin-audio', version: 1, src, type: 'python'|'abc'|'zzfx'|'sfxr'|'pattern', settings: { sampleRate, channels, scene, renderer }, files }.

---

8. Sibling conventions and conflicts (flagged, not silently resolved)

• Newest sibling: Livelier (2026-10-29) is newest by feature work. It still lives at

curtcox.github.io/livelier, is not in OWNED_APPS, and uses its own window bootstrap with 'unsafe-eval'. The B13 migration commits on Gnuplot, Manim, Python, Pages, sheets and modl (2026-10-30) are newer for hosting and domain conventions. I'll follow B13 for hosting, migration and legacy notices, and Gnuplot's layout as the brief directs. Audio Online does not need 'unsafe-eval'.

• Document format: Gnuplot saves {format:'hashbin-diagram', type:'gnuplot', gnuplot:{…}}, but

the brief specifies hashbin-audio with a settings key. I'll follow the brief.

• Offline: Gnuplot's service worker does its own hash checks and does not use

hashbin-cache-v0.4.0.js. ManimOnline, PythonOnline and Livelier do. The brief asks for the vendored cache helper, so I'll follow Manim/Python. The script name stays Gnuplot's scripts/update-offline.mjs (Manim's is update-offline-hashes.mjs; Livelier uses npm run offline).

• Pyodide pins differ: Manim 0.27.0, sheets 0.27.7, Python 314.0.7. I'll use 314.0.7. Siblings

serve Pyodide from a pinned jsDelivr URL plus a verified cache rather than committing it, which conflicts with the brief's "vendored WASM is committed" (D6).

• Stale sibling doc: Gnuplot's README still gives the old GitHub Pages return URL in one section

and gnuplot.hashbin.page in another.

• Firefox locally: Playwright's Firefox fails to launch on macOS 27, matching hashbin.org's note.

Desktop Firefox through WebDriver BiDi works and would be the local recipe. CI on Linux keeps the bundled Firefox.

• Sandbox hardening: the SDK prelude doesn't remove storage APIs, and WebKit exposes caches

(opaque partition) to the worker. I'll copy PythonOnline's harden() step.

---

9. What happens after sign-off (unchanged from the brief, adjusted for the above)

1. Skeleton from Gnuplot conventions, plus the deterministic renderer, WAV export and the three-engine

CI, with one path end to end: Python → timeline → WAV.

1. Notations (ABC, ZzFX, sfxr, pattern), then Pico, then effects/tracks/stems, then neural voices on

demand (D3/D4).

1. Gallery and covers.
2. hashbin.org registry entry and staging.

Stopping here for sign-off on D1–D7.

---

10. Sign-off (2026-10-30)

Curt approved every recommendation:

• D1: render only in the sandbox worker, using original deterministic DSP. Encode on the editor origin.
• D2: SVOX Pico is the bundled voice. The lingware caveat is accepted.
• D3: write an original CMUdict-based G2P, with no GPL phonemizer.
• D4: Kitten TTS nano and Kokoro-82M are available on demand.
• D5: v1 accepts WAV uploads only.
• D6: serve Pyodide like PythonOnline, from a pinned CDN with a verified offline cache.
• D7: use a placeholder app ID until Curt supplies the real one.

---

11. Corrections found while building Step 3 (2026-10-30)

The Step 1 plan held, but four findings changed details. Each one has a test.

1. ZzFX also needs a source rewrite, not only a Math shim. It uses the ** operator in four

places. No prelude can intercept **, and engines evaluate it differently. That was the real cause of the boom mismatch in §2.2. The loader rewrites exactly those four expressions to Math.pow and refuses any file whose hash or ** usage differs. The vendored file stays byte-identical.

1. Pico keeps state between utterances. It is held in C globals that pico_resetEngine and even a

new engine leave behind, so one utterance changed the next. The WASM heap is now restored to its just-loaded snapshot before every utterance, as Gnuplot Online does around each run.

1. jsfxr 1.4.1 emits NaN as its first sample with the platform Math too. Non-finite samples

become silence.

1. WebKit's AAC encoder returns an ES descriptor as its "description". Chromium returns the bare

AudioSpecificConfig. The M4A muxer handles both.

Two small additions:

• Notation documents get a transparent −1 dBFS master limiter, because sfxr's explosion preset

exceeds full scale.

• Ducking keys on the other track after its effects and before its pan.

---

12. Realignment with the brief (2026-10-30)

A review against the original brief found these gaps after Step 3, and closed them:

• §2 sources: Sample("file.wav") (uploaded files, with an Audio files dialog) and

Sample(cid="…") were missing. Both are now in, as decided under D5. CID samples are fetched and SHA-512-verified on the editor origin and handed to the sandbox as bytes; inline CIDs decode in the sandbox.

• §7 proposal leftovers: these are now in:

- clip .speed() (tape-style resampling) and .pan(); - .loop(until=); - Envelope(curve="linear"|"smooth"|"exp"); - Peak, LowShelf and HighShelf. The proposal said Shelf; two names are clearer than a mode argument.

• Step 4:

- five original examples were added: door, sound-design hallway, CID wavetable, uploaded loop and neural narrator, for 16 in all; - gallery.html with covers and lazy previews; - scripts/render-gallery.mjs, whose --check mode is part of npm run check.

Previews are AAC (.m4a), which Safari, Chrome and Firefox play (Firefox through the operating system's AAC decoder). Encoding needs Chromium's platform AAC encoder (macOS/Windows), so CI checks fingerprints and does not re-encode.

• Step 5: hashbin.org registers and stages audio with the placeholder app ID. Two findings:

- Staging must rehash the offline manifest after it writes the configured migration module, as it already does for Pages. Otherwise the hosted service worker refuses that file. - The legacy notice is deliberately not installed yet. It would announce a move to audio.hashbin.page before that site exists.

One latent bug surfaced while testing: Pyodide caches imported modules, so a second render with a different engine object kept calling the first render's _audio_js bridge. The worker always reuses one engine and was unaffected; Node rendering with per-render samples was. AudioPython.run now drops the cached module each run. All existing golden hashes are unchanged.