Reading a Negative Result

01What is actually generating the audio

Before anything else: the model does not assemble music out of parts. It sculpts it out of noise.

Start with 30 seconds of static — pure randomness, no structure. Now ask a question: if this static were slightly less random, and slightly more like "smoky late-night jazz trio, upright bass, brushed drums," what would it look like? Apply the answer. Then ask again, of the result. Repeat a fixed number of times. What is left at the end is audio.

That process is called diffusion, and each pass is a step. Our runs use eight of them.

Three details matter for reading our results.

First, the model does not work on the waveform directly. It works inside a compressed representation called a latent space — audio is encoded going in, decoded coming out. All the sculpting happens in that compressed middle.

Second, the randomness is controllable. The starting static is drawn from a seed; we fix ours at 4242. The rule deciding how much noise comes off at each step is the sampler, and how hard the model is pushed toward your text is the CFG scale. Hold all of those fixed and the process is deterministic — we confirmed this by re-rendering one condition four weeks later and getting a byte-identical file.

Third, and this is the one to carry forward:

So when we say we wanted the model to follow a chord chart, we were never going to do it by asking. There is nothing in there to ask.

02Conditioning, and why it is the whole ballgame

Conditioning is any input that steers generation. The catch: a model only listens through pathways it was trained to listen through.

The familiar pathway is text conditioning — you write a prompt, it nudges each step. But text is a blunt instrument for structure. You can ask for "a twelve-bar blues in F" and get something that sounds like one without ever landing on the right chord in the right bar.

The other pathway our model exposes is audio conditioning, passed as a parameter named init_audio. Instead of starting from pure static, you hand the model a piece of audio, it adds noise to that, and denoising starts from there. Your audio is the starting shape rather than randomness.

It is worth pausing on why that does anything at all. If you add noise to a sound and then take noise away, why do you not simply get your sound back — or something completely unrelated?

Because noise does not bury a signal evenly. It swallows fine detail first and the loudest, coarsest structure last. A little noise hides the character of the instrument while the broad shape of the sound still shows through; a lot of noise erases everything down to nothing. So when denoising resumes, whatever was still visible under the noise gets rebuilt and reinforced, and whatever was buried gets invented from scratch.

The one number you need: init_noise_level

How much noise gets added to your audio before denoising begins is set by init_noise_level. It runs from 0 to 1, and it is the single most important dial in this entire experiment.

Two more pathways exist and are worth naming, because they are what a solution would look like. Inpainting regenerates a masked region in place. A control branch — the idea popularised as ControlNet in image models — is a separately trained input that accepts a structured signal directly.

We did not have a control branch. We had init_audio. And that distinction is the entire experiment, because of this:

We rendered our chord chart as a simple synthesiser part — we call it the skeleton — and passed that. If the output then matched the chart, two explanations fit equally well: the model understood the harmony, or the model kept our synthesiser. Everything that follows is an attempt to separate those.

03The measurement instrument

To ask whether the output matched the chart, we need something that listens to audio and names chords. That instrument has properties, and they matter as much as the results.

Ours works on chroma. Take a slice of audio, measure how much energy sits at each of the twelve pitch classes, and ignore which octave they came from. The standard tool for this is a constant-Q transform; ours uses a plain FFT binned to the nearest pitch class, which is simpler and equivalent in spirit.

Chroma is deliberately blind to timbre. A piano and a saxophone playing the same chord give nearly the same chroma vector. For naming harmony that is exactly the property you want. Hold onto it anyway — it becomes the trap.

To turn a chroma vector into a chord name we use template matching: build an idealised twelve-slot pattern for each chord in the chart, and score the measured vector against each one using cosine similarity. Highest score wins.

Which brings us to the property that governs how every number on this page should be read. Our grader is a forced choice over exactly six chords — the six that appear in our chart — across twelve bars.

Twelve bars times a one-in-six chance is two. An instrument that understands nothing at all scores 2 out of 12 on this chart. That is not a bad score. It is the floor — the chance baseline.

Every 2/12 you encounter later means "indistinguishable from guessing," never "performed poorly."

Two caveats, both of which make the true chance level a little different from that clean 2 — and neither of which we measured. Our grader accepts a near-miss within 5% of the winner, which is more generous than a strict one-in-six. And bar 8 contains two chords, both of which must be right, which is harsher. The honest position is that we know chance is around 2 and we never established it properly. What you would want is the full null distribution: not just the average guess, but how often chance alone would produce a 4, or a 6.

What chroma cannot see

Because chroma folds away octaves and timbre, several distinctions vanish. Enharmonic pairs are identical to it by construction. Octave collapse means a chord in root position and the same chord inverted look the same. C6 and Am7 contain the same four pitch classes and cannot be told apart at all.

And critically, our grader is unvalidated. We never ran it on recordings where the right answer was independently known. We do not know its error rate. Every number it produced should be read with that attached.

04How you tell a result from an artefact

This is the part that transfers. Nothing here is specific to audio.

The problem in our experiment is a confound. We changed the harmony and the audio at the same time, because the audio was how we passed the harmony. When your input survives into your output and your instrument then measures it, that is leakage, or bleed.

Before running anything we wrote down what would count as failure. That is pre-registration, and the specific rules are kill conditions. One of ours read: if accuracy holds at high noise, where the skeleton is acoustically gone, the effect is real conditioning.

Then two controls. A negative control: we rebuilt the skeleton with a deliberately alien timbre — odd harmonics, sustained rather than plucked, a different bass — while copying the chord voicings across verbatim. Same harmony, different sound. If the model were reading harmony, swapping the instrument should change nothing.

And a mismatch test: grade a clip against the wrong chart and check that it fails at exactly the bars that differ.

The ablation here is the noise sweep itself. Turning init_noise_level up progressively removes the skeleton. If accuracy came from the harmony, it should survive. It did not: at 0.9 both skeletons score 2 of 12, which is chance.

Which leads to the distinction worth naming, because it is the specific way this class of experiment goes wrong:

Two words that are not synonyms

Re-rendering one condition four weeks later produced a byte-identical file. That establishes determinism, and it is genuinely useful: the pipeline has no hidden randomness. It does not establish reproducibility, which would require someone else confirming the finding independently. Running the same thing twice is not replication.

And every cell in our table is n = 1. One render per condition, one seed. That constrains how hard any single cell can be pushed.

The table, in full

Accuracy and leakage move together at r = +0.98. Accuracy against distance-from-skeleton runs at r = −0.97. Read those alongside one more measurement: the two skeletons differ from each other by 0.507, while at noise 0.3 the generated output sits 0.052 from the skeleton that produced it. The output is closer to its own input than the two inputs are to each other.

One loose end, stated as loosely as it deserves. At noise 0.7 the scores are 6 and 4 against a chance level of about 2, at a point where most of the skeleton is gone. That might be a weak real effect. It is a hypothesis, not a finding — and it is unresolved for a specific reason: with no null distribution, we cannot say how often chance alone would hand us a 6.

05Symbols versus audio

The confound was not bad luck. It followed from the representation we had available.

A chord can be held two ways. As a symbolic object it is a label with an exact definition; checking whether two symbols match is trivial and cannot be wrong. As audio it is a continuous signal, and any label you attach must be estimated by an instrument that can be mistaken.

Our chart was symbolic. Our conditioning pathway accepted only audio. So we rendered symbols into sound, and at that moment the harmony stopped being true by construction and became something we had to measure — with an instrument sharing a channel with the input.

A pathway that accepted symbols directly would not have this problem. The control signal and the measurement would no longer travel on the same wire, and the confound would be structural rather than something you argue about afterwards. That is what a trained control branch is for.

We have not built one, and nothing here shows that it would work. It is the direction the next experiment points, not a result.

—What is missing

Honest gaps, cheapest first.

1. The null distribution — and it needs no GPU Grade the clips we already have against a few hundred shuffled chart labels. That yields the real chance distribution, which tells us whether the 6 at noise 0.7 means anything. It is the single cheapest unrun experiment here, and the most load-bearing.
2. Validate the grader Run it on material whose chords are known independently. Until then its error rate is unmeasured, and every number it produced inherits that.
3. A metric that hears the bass Chroma cannot separate C6 from Am7, or a chord from its inversion. A register-sensitive second metric would cover the blind spot.
4. The mismatch test at every noise level We ran it at one setting. It should run at each, to show the localised failure dissolving as the skeleton does.
5. Replication beyond n = 1 One render per cell, one seed. Repeats across seeds would give each number an error bar.