Section 01 · Attribution MethodologyReading a Negative Result in Audio Conditioning
Before training a single adapter weight, we have to separate genuine harmonic conditioning from input signal leakage.
A diffusion audio model does not assemble music out of discrete tracks. It starts with static drawn from a deterministic seed (fixed at 4242 in this test) and runs a learned reverse diffusion pass across eight steps inside a 64-channel latent space.
One generation · eight steps, noise to audio
The model never adds a bass line to a drum track. Every part of the output emerges at once, getting less noisy together.
Why init_audio creates a measurement confound
When you steer a diffusion model with an audio file via init_audio (audio conditioning), the dial that governs how much of your input survives before denoising begins is init_noise_level (from 0.0 to 1.0).
How much of your audio is left before generation starts
At 0.3 your audio largely survives and the model lightly resurfaces it. At 0.9 almost nothing survives and the model generates nearly from scratch.
Because audio carries harmony and timbre fused on the same wire, passing a synthesized chord chart (the skeleton) into init_audio and grading the output with a chroma template matcher creates a textbook leakage confound. We ran a pre-registered ablation across two orthogonal skeletons (a plucked toy synth and a sustained odd-harmonic alien synth playing identical voicings) across four noise levels.
Interactive Attribution Explorer · Skeleton Bleed vs. Harmony Accuracy
Four measured conditions · Seed 4242 · No interpolationConditioning skeleton, as it survives into the output
Harmony accuracy, bars matched of 12 (vs. 2/12 chance floor)
- Toy skeleton
- 12 / 12
- Alien skeleton
- 12 / 12
- Distance to skeleton
- 0.052
- Chance floor
- 2 / 12
When your control signal and your measurement instrument share a channel, you can measure your own input and mistake it for a result.
Hear the Attribution Confound (30-Second Synchronized Stems)
Play the dry THIRI skeleton, then the conditioned diffusion output at noise 0.5, then the unconditioned text-only baseline. Switching stems preserves bar position.
The conditioned clip sits 0.078 from the skeleton that produced it. The two skeletons sit 0.507 from each other. The output is closer to its own input than the inputs are to one another.
| Condition | Noise | Bars Matched | Bleed Index | Dist to Skeleton | Dist to Text-Only |
|---|---|---|---|---|---|
| toy | 0.3 | 12 / 12 | 0.849 | 0.052 | 0.294 |
| toy | 0.5 | 12 / 12 | 0.783 | 0.078 | 0.281 |
| toy | 0.7 | 6 / 12 | 0.277 | 0.348 | 0.133 |
| toy | 0.9 | 2 / 12 | 0.136 | 0.380 | 0.060 |
| alien | 0.3 | 12 / 12 | 0.733 | 0.093 | 0.254 |
| alien | 0.5 | 10 / 12 | 0.674 | 0.119 | 0.245 |
| alien | 0.7 | 4 / 12 | 0.336 | 0.317 | 0.160 |
| alien | 0.9 | 2 / 12 | 0.135 | 0.391 | 0.061 |
An earlier writeup reported 12 of 12 bars as evidence the model followed a computed harmony layer. Because accuracy correlates with input bleed at r = +0.98 and collapses to 2/12 (chance) at noise 0.9, teaching or steering harmony requires either adapter training inside the DiT blocks or timbre-decoupled latent intervention. Sections 02 through 05 report our measurements on both fronts.
Section 02 · Native C++/ggml DiT Training EngineBenchmarking 8 Adapter Families (LoRA, DoRA, BoRA & SVD -xs) Across CUDA, Metal & Vulkan
To test adapter architectures without Python/PyTorch overhead or cloud queue variance, we built sa3-train and sa3-train-web natively in C++/ggml (sa3.cpp d7939a2, ggml v0.16.0). It trains lora, dora-rows, dora-cols, bora, and their SVD-initialized -xs variants directly against medium-base and small-music-base F16/Q4_K_M DiT weights.
Interactive Adapter Architecture Explorer · All 8 Families on Stable Audio 3 Medium-Base
RTX 5070 Laptop GPU · F16 Base · Rank 16 / Alpha 16 · 512 Frames (47.6s) · Steps 10-40 Meandora-rows (Weight-Decomposed Low-Rank Adaptation)
Decomposes every targeted DiT Linear/Conv weight into a trainable F32 row-wise magnitude vector m and a directional update W0 + (alpha/r) B A. Step 1 computes a one-off base_norm_sq reduction (6.17 s), after which steady steps run at 1.25 s/step.
From 92.76 s/step (24.6 GiB) down to 1.95 s/step (1,126 MiB)
Six of the eight adapter families initially ran a monolithic backward pass requesting 24.6 GiB of allocator memory on an 8 GiB GPU, paging to system RAM (`92.76 s/step` on dora-cols). By materializing effective weights inside each segment's checkpointed backward graph and interposing a contiguous tensor view (ggml_cont) before column-norm scaling, allocator footprint dropped to 1,126 MiB and step time fell 47.5x to 1.95 s/step.
--lora-scope core Saves 10.6% Step Time at Identical Loss
full adapts all 228 DiT Linear/Conv weights (`82.3 MB`). core drops `local.{0,2}` and head/tail projections (`11.6%` of parameters: `19.06M` kept vs `2.51M` dropped), adapting only the 7 per-block projections (`self.qkv`, `self.out`, `cross.q`, `cross.kv`, `cross.out`, `ff.proj`, `ff.out`) across 24 blocks (`168` targets, `72.7 MB`). Across 2,000 updates on 23s crops, the loss curve stayed within 0.0013 (`0.8843` vs `0.8830`) at `0.642 s/step` vs `0.718 s/step`.
Excluding seconds_total & Disabling Path Captions
sa3-train intentionally excludes the 229th seconds_total conditioner Linear so small datasets cannot hijack duration conditioning. It also defaults use_paths=false: training on raw filenames like 03 - Gettysburg [Q-gtqz_il-w].wav teaches the adapter to emit memorized audio on noise tokens, diluting text conditioning and breaking CFG dropout (`0.1`).
Cross-Backend Hardware Benchmarks · Native C++/ggml vs. PyTorch 2.7.1 & Apple MLX
Every row below was measured on identical pre-encoded Ratatat latent tensors (1 latent frame = 4096 samples at 44.1 kHz = 0.0929s; 512 frames = 47.6s, 256 frames = 23.8s), batch size 1, seed 42, and dora-rows rank/alpha 16.
| Backend & Implementation | Model & Crop | Targets | T5 Cond | DiT Fwd+Bwd | AdamW | Total / Step | 2,500 Updates | Peak RSS / Mem |
|---|---|---|---|---|---|---|---|---|
| sa3.cpp CUDA (RTX 5070 Laptop) | medium-base F16 · 512f (47.6s) | 228 | 8.2 ms | 986.2 ms | 69.2 ms | 1,065.4 ms | 44m 24s proj. | 1.10 GiB alloc |
| PyTorch 2.7.1 + CUDA 12.8 (FlashAttn) | medium-base F16 · 512f (47.6s) | 229 | n/a | n/a | n/a | 1,775.0 ms | 74m 02s obs. | ~7.8 GiB VRAM |
| sa3.cpp CUDA Validated Full Run | small-music F16 · 512f (47.6s) | 192 | ~6.0 ms | ~448.0 ms | ~31.0 ms | 485.0 ms | 16m 10s (2,000u) | 1.78 GiB peak |
| sa3.cpp Apple M4 Metal (32x16 SIMD Tile) | medium-base F16 · 512f (47.6s) | 228 | 29.8 ms | 8,574.5 ms | 43.0 ms | 8,648.8 ms | 6h 00m (5h 29m / 2k) | 5.75 GiB RSS |
| sa3.cpp Apple M4 Metal (Scalar OUT_PROD) | medium-base F16 · 512f (47.6s) | 228 | 35.3 ms | 22,273.8 ms | 53.8 ms | 22,364.3 ms | 15h 32m proj. | 5.76 GiB RSS |
| gary4local Apple M4 MLX (Matched 228) | medium-base F16 · 512f (47.6s) | 228 | n/a | n/a | n/a | 7,285.0 ms | 5h 04m proj. | 16.27 GiB RSS (23.8G peak) |
| sa3.cpp Vulkan dGPU (RTX 5070 CoopMat) | small-music F16 · 256f (23.8s) | 192 | 12.6 ms | 666.0 ms | 34.2 ms | 714.2 ms | 29m 46s proj. | 1.52 GiB alloc |
| sa3.cpp Vulkan Intel iGPU (UMA) | small-music F16 · 256f (23.8s) | 192 | 100.2 ms | 5,383.8 ms | 33.2 ms | 5,518.6 ms | 3h 50m proj. | 1.78 GiB UMA |
| sa3.cpp CPU (Core Ultra 9 275HX, 24T) | small-music F16 · 256f (23.8s) | 192 | 115.4 ms | 13,990.8 ms | 59.2 ms | 14,166.8 ms | 9h 50m proj. | 2.28 GiB RAM |
Text-Encoder Quantization (F16 vs Q8_0) Is 0.84% of a Step
As a GPU heats across a benchmark session, step time drifts monotonically upward (1062 ms in block 1 to 1064 ms in block 2, 1075 ms in block 3, and 1080 ms in block 4). Running A-then-B assigns all thermal drift to condition B. Under counterbalanced ABBA ordering (F16, Q8_0, Q8_0, F16), pooled F16 is 1069.5 ms (9 ms T5) and Q8_0 is 1070.0 ms (8 ms T5): a +0.05% difference. Pick --t5-encoding q8_0 or --evict-text-encoder (-285 MiB) strictly for RAM/VRAM headroom, never for speed.
285s Down to 12s Startup + Trajectory-Safe Checkpointing
--latents-cache fingerprints the autoencoder weights, --target-latent-rms (0.9), and decoded audio SHA-256, cutting second-run startup on the 10-track Ratatat set from 285 s to 12 s with bit-identical loss trajectories. Every checkpoint writes an immutable pair: a lean adapter-step-N.gguf for sa3-generate --lora and a trainer-state-step-N.gguf sidecar holding AdamW moments, RNG state, and dataset cursor.
Section 03 · 8 Pre-Registered Empirical Runs (SA-01 to SA-08)Where Diffusion Succeeds, Where It Hits a Cliff & What a $1.20 Rank-8 LoRA Learned
Across eight pre-registered experiments, we tested Stable Audio 3 (`sa3.cpp` local CPU + HF Space + `fal.ai` LoRA trainer) on continuation, surgical inpainting, loop slicing, text prompt obedience, melody restyling, post-hoc Csound harmonic forcing, a 7-section hybrid arrangement, and our first 128-clip chord adapter pilot.
| Run ID | Experiment & Question | Control & Noise Floor | Measured Ground-Truth Result | Architectural Takeaway |
|---|---|---|---|---|
| SA-01 | Extend (8s to 12s / 20s) Can SA3 continue an 8s G phrygian pad? |
8s source pad (seed 11); scale floor 56.9% |
Kept 0-8s matches (r = 0.993), then drops into a 1s silent hole (-61.2 dB on 12s; -45.3 dB on 20s; -48.7 dB medium) and adds out-of-key B natural (3.8%). |
Outpainting from a hard boundary at 8.0s resets the latent energy envelope and drifts out of mode. |
| SA-02 | Inpaint (Repair 2s) Can SA3 rewrite a bad half-bar in place? |
Unmasked region; G phrygian floor 56.9% |
Outside 3-5s untouched (r = 0.993), inside rewritten (r = 0.021). Repairing a half-bar with stray F# (23.4%) lifted G phrygian share 55.4% to 85.3% (90.7% full loop). |
Bidirectional context from left + right neighbors constrains pitch classes far better than open continuation. |
| SA-03 | Loops (Bars 2-5 Crop) How do you get a seamless 4-bar loop? |
Direct 4-bar request (10.67s); C minor floor 56.9% |
Direct 4-bar take writes an ending (last 50 ms at -95.7 dB, 63.5% in key). Asking for 24s and cropping bars 2-5 yields 470,400 samples, 16/16 beats (4.8 ms err), and 90.9% scale share. |
Always generate long (24s) and harvest interior bars (2-5) with a 2 ms boundary crossfade. |
| SA-04 | Prompt Words Sweep Which musical words does SA3 obey? |
No-key baseline (k0-none); C#m floor 59.5%, chord floor 33.1% |
Obeys BPM (20/20 at 90/180 BPM) and Root (C#m share rose 39% to 67% HF, 36% to 58% local). Ignores Mode (C#m read Major 4/4 times; G phrygian 84% vs 85% none) and Chords (2/12 bars). |
CLAP/T5 text tokens steer tempo and root center, but have zero control over modal quality or bar-by-bar chords. |
| SA-05 | The Restyle Cliff How far can SA3 restyle a Csound melody? |
16-note Csound G phrygian melody (490 Hz centroid) |
Keeps 16/16 notes at noise 0.3, 0.4 (r = 0.91-0.95), 13-14/16 at 0.5 (99.5% scale), then drops off a cliff to 2-5/16 notes at 0.6-0.8 (r = 0.01) as it converges to its training prior. |
Single-timbre VAE latents entangle pitch and timbre: breaking the Csound timbre cage at noise 0.6 erases the notes. |
| SA-06 | Forcing Chords Afterwards Can Csound lock Gm7 | Ebmaj7 | Cm7 | D7 on Side B? |
Untreated SA3 (19-77%); tritone 8-15%; chord floor 33.1% |
Csound pvsvoc (2048/256): 86%-98% every bar across all 3 takes (vox D7 bar jumped 19.2% to 96.6%; 75% wet kept 90-97%). Resonator (resonz Q=60): 84-99% except vox D7 (65.3%, source had 0.4% A). |
Passive filters (resonz) cannot ring missing leading tones; active Csound phase vocoding (pvsvoc) builds them. |
| SA-07 | The Hybrid Beat Csound writes notes, SA3 restyles at 0.50-0.55 |
Text-only SA3 pad (73%) & vox (71%); scale floor 56.9% |
76.67s, 90 BPM 7-section master holds 90%-94% on G phrygian across all sections (restyled pad 97.0%, flute 97.8%, vox 89.7%, Csound bass 91.3%). |
Operating right at the 0.50-0.55 cliff edge delivers real instrument ceiling (>90%) across a full arrangement. |
| SA-08 | Chord Adapter Pilot Can a $1.20 rank-8 LoRA teach minor vs. major? |
Base SA3 (+0.17 gap, 4/8 pairs); target ≥+0.40 & 6/8 |
Clean negative result: Held-out root minorness gap rose +0.17 to +0.24 (5/8 pairs vs 6 needed). Triad share rose 67% to 74% and C#m EP jumped 0.05 to 0.52, but root priors on Eb, F#, B won. |
A 400-step rank-8 text adapter nudges attention in the right direction but cannot overcome deep DiT root priors alone. |
Section 04 · Theoretical + Empirical SynthesisSolving the 0.5 Restyle Cliff & the D7 Leading-Tone Trap With Csound Latent Basis Probing
Runs SA-05, SA-06, and SA-08 exposed two structural walls in audio diffusion: single-stem VAE latents entangle pitch with timbre at the 0.5 to 0.6 restyle cliff, and passive Side B resonators stall at 65.3% when a modal take lacks leading tones (0.4% A and 4.0% F# on Bar 4 D7). Because Csound compiles deterministic physical models at 492x real-time, we use Csound as both a Side A multi-timbre latent basis probe and a Side B phase-vocoder harmonic lock.
Decoupling Pitch From Timbre Across the 0.60 Cliff
Why did SA-05 drop from 13/16 notes at 0.50 to 5/16 at 0.60? Because a single Csound stem's latent z_cs pins the spectral centroid to 490 Hz. By rendering the exact same THIRI voicing (thiri_voice --voicing jazz4: F3 D4 G4 Bb4 | Eb3 D4 G4 Bb4 | Eb3 C4 G4 Bb4 | D3 C4 F#4 A4) across M = 6 to 8 orthogonal Csound instruments (sine, FM foscili, bowed waveguide wgbow, plucked string wgpluck2, formant pulse fofilter) and averaging their VAE encodings:
mu_harm^(b) = (1/M) * Sum_m E_VAE(x_cs,m^(b))
Instrument-specific formants cancel out while fundamental and overtone phase survive. Pairing mu_harm^(b) with a counterfactual leading-tone vector v_lead^(b) = E_VAE(D7: F#4+A4) - E_VAE(Fm7: F+Ab) steers the DiT across the 0.60 boundary without locking the output to a sawtooth timbre.
Synthesizing Missing Leading Tones & Healing Vocoder Sheen at 0.25
In SA-06, the G-phrygian vocal chops contained only 4.0% F# and 0.4% A in Bar 4, causing resonz (Q=60) to stall at 65.3%. Csound's pvsvoc (fftsize=2048, hop=256, pre-shifted 2560 samples) imposes the vocal formant envelope onto a band-limited Csound carrier, jumping Bar 4 from 19.2% to 96.6% (and 90%-97% at 75% wet blend vs 33.1% floor and 8%-15% tritone foil).
x_master = D_VAE( DiT( E_VAE( Blend_0.75(pvsvoc(x_sa, x_cs), x_sa) ), tau=0.25 ) )
Passing that 75% wet pvsvoc stem back through Stable Audio at tau = 0.25 (safely inside SA-05's 16/16 note-retention plateau) replaces synthetic vocoder sheen with natural acoustic air while preserving every chord tone.
Section 05 · Pre-Registered Next Experiment Matrix (SA-09 to SA-12)Testing Multi-Timbre Basis Init, Side B Latent Heal & Native DoRA-Rows-XS Adapters
Every upcoming run is pre-registered below with its exact control, empirical chance floor (33.1% 4-note chord / 56.9% 7-note scale), and quantitative pass/kill bar before execution.
| Run ID | Experiment & Hypothesis | Engine & Adapter Config | Control & Floor | Pre-Registered Pass Bar |
|---|---|---|---|---|
| SA-09 | Crossing the 0.60 Restyle Cliff Does a 6-timbre Csound latent centroid keep melody notes across tau = 0.60 & 0.65? |
Csound 6-timbre probe bank (sine, FM, wgbow, wgpluck2, fofilter, organ) + SA3 img2img | Single-timbre SA-05 at 0.60 (5/16 flute, 2/16 trpt); scale floor 56.9% |
At tau = 0.60, notes kept rises from 5/16 to ≥ 12/16 while spectral timbre shift exceeds ≥ 10.0 dB. |
| SA-10 | Side B pvsvoc + 0.25 Latent Heal Does a tau = 0.25 / 0.35 VAE pass remove vocoder sheen while keeping the D7 lock? |
Csound pvsvoc (2048/256, 75% wet, -2560 sample shift) + SA3 at tau = 0.25, 0.35 |
Raw SA3 vox D7 (19.2%), raw pvsvoc (96.6%), tritone (13%), floor 33.1% |
Every bar holds ≥ 80% chord share (vox D7 ≥ 75%) and spectral distance to natural SA3 vox improves by ≥ 4.0 dB. |
| SA-11 | Inference LoRA Scale + Root-5th Init Does alpha = 1.5, 2.0 + a tau = 0.65 Csound root-5th drone break held-out root priors? |
sa3.cpp local CPU + lora-chordpilot-f32.gguf at alpha = 1.5, 2.0 ($0 training cost) |
Base SA3 (+0.17 gap, 4/8) & SA-08 alpha=1.0 (+0.24 gap, 5/8 pairs) |
Held-out minor minus major gap (C#, Eb, F#, B) reaches ≥ +0.40 and minor beats major in ≥ 6/8 pairs. |
| SA-12 | Native DoRA-Rows & SVD -xs Harmonic Adapter Does rank-16 dora-rows / dora-rows-xs on multi-timbre Csound clips beat rank-8 LoRA? |
sa3-train native C++/ggml · dora-rows & dora-rows-xs · rank 16 · --lora-scope core (168 targets) · 2,000 steps |
SA-08 rank-8 400-step cloud LoRA (+0.24 gap, 5/8 pairs); seconds_total excluded |
Text-only inference on held-out roots reaches ≥ +0.45 minorness gap, ≥ 7/8 pairs, and ≥ 78% triad share. |
Cross-Architecture Validation · Symbolic Injection (Magenta RT2 / Lyria) & Closed-Loop Vocal Calibration (ElevenLabs)
To verify that our harmonic referee (thiri_detect.py, 24 deterministic rules) generalizes beyond diffusion models, we ran the same pre-registered protocol across autoregressive token models (Google Magenta RT2, ImprovRNN), latent sequence VAEs (MusicVAE), streaming audio models (Lyria RealTime), and speech-cloned singing voices (ElevenLabs IVC).
From 39%-42% Unsteered to 71%-99.9% With Explicit Pitch-Class Steering
- Magenta RealTime 2 (Run E3): The
guidance_weight/cfgdial is a harmonic no-op on bare chord symbols (42%energy vs33.1%4-note floor). Passing one voice-led THIRI voicing per bar as explicit notes lifts chord-tone share to 71% across 32 bars (and83%in Run 3E1). - ImprovRNN Logit Nudge (Run E1 & Run 3 E4): Unsteered improvisation hits the harmonic color tone only
13.1%of the time; applying a+4THIRI logit nudge on theF#leading tone ofD7lifts hit rate to 78.6% (andF#share onD7from55%to75%). - MusicVAE Latent Morphing (Run E2): Slotting THIRI endpoints into latent interpolation preserves 99.9% contour agreement across 7 rungs vs
0%on a mirror-image negative control.
From 1,766.9 Cents Unconstrained Drift Down to 11.8 Cents Median Error
- Speech-to-Singing Drift: Asking an unconstrained speech clone to sing a jazz standard produces up to 1,766.9 cents of pitch drift.
- Closed-Loop YIN + Formant Lock: Measuring fundamental frequency (
f0) against a deterministic Csound guide track and applying formant-preserving pitch calibration brings median pitch error down to 11.8 cents (p95 = 31.78 cents). - 4-Part Ellington SATB Choir: Using the calibrated pipeline, four speech clones (
Lily, Sarah, Dennison, Charlie) sing drop-2 voicings on In a Sentimental Mood and an 80-bar blues master (Fly Anyway,93.75 BPM,-16 LUFS).
Section 06 · Research Dispatch & Open Engineering LabFollow the Benchmarks or Run the Harnesses Yourself
We publish our negative results, adapter training benchmarks, and Csound x diffusion harnesses openly. No sales sequences and no gated whitepapers.
Get the Lab Notes When a Benchmark Breaks
Game THIRI is our public dispatch for engineering leaders, researchers, and builders. We write when we complete a pre-registered run (SA-09 through SA-12), ship a native sa3-train kernel optimization, or publish a negative result. Nothing auto-sends.
Run the 22 AI Music Skills & Harnesses in Skool
Every experiment on this page is packaged as a reproducible Claude Code and Antigravity skill inside our Skool community, including thiri-stable-audio, csound-instrument, thiri-detect (24 deterministic harmony, audio, and brand rules), and our sa3.cpp training scripts.
- T.H.I.R.I. Builders Lab: Weekly live architecture & experiment reviews, full
HARMONY.jsoncontracts, Csound.csdorchestras, and GGUF adapter recipes. - 12 Free Working-Musician AI Skills: Stage 1 Facts Sheet, P0 Attribution Checklist, Noise-Sweep Controls, and Closed-Loop Pitch Auditing.
- Full Source & Monograph: Download the 4-page PDF Monograph and inspect the raw
facts.jsonreceipts.