What we have learned
Twelve lessons, each tied to its labs.
Each lesson cites the labs it comes from. All are from our own runs, at the scopes stated in those labs.
-
Lesson 1: Check the computation, not just the text it produced.
Identical printed losses hid different checkpoint bytes. An altered RMSNorm order changed the digest and left the 16 tokens alone. Edited embedding rows hid from output checks that no prompt exercised.
From the labs
- L06One flipped bit in the weights: which checks notice?, Positive
- L07Realistic edits to the embedding table, including one hidden row, Positive
- L15E25: the RMSNorm multiply order matters, and our spec sentence saying otherwise was false, Positive
- L09Ordinary logging versus replay-based receipts on one-field edits, Positive
- L12Training checkpoints across CPUs and instruction sets, Mixed
-
Lesson 2: In our fp32 tests, the pinned order of operations determined the bits.
The same order matched across the CPUs and the one GPU we tried, and a different order broke the digest; default library kernels did not match across two CPUs in training. Heavily vectorised builds matched when the order was preserved. Scope: one small model, a few machines, run by us.
From the labs
-
Lesson 3: A spot check can give a comfortable wrong answer.
4,096 samples per binade never hit the worst points among about 8.4 million. A maximum measured on one prompt was only that prompt's maximum, and widening the sweep corrected our own published number.
From the labs
-
Lesson 4: Trying to break our own work finds the holes.
A mismatch exposed a spec error, a mis-filed receipt exposed a verifier hole, and exhaustive tests exposed a defect in a spec guard. Each was recorded in the open.
From the labs
- L04Clean-room reimplementations: one mismatch (from a memo), one match (from spec v0.1), one partial that exposed a spec error, Mixed
- L08Receipt-verified agent evaluation, and the hole we found in our own verifier, Mixed
- L14E24: the spec never says which Unicode version defines a letter, Mixed
- L17E27: exp, ln and SiLU on every float32 input, and a defect in our spec, Mixed
- L21E41: the spec pins how text becomes tokens, but not how tokens become text, Negative
- L13E22: which spec rules does the test vector actually exercise?, Mixed
-
Lesson 5: Measure the machine and the shipped binary before measuring the code.
A throttled CPU and a compiler setting each invalidated speed numbers while everything appeared to work; a soft-float build explained one separate slowness and is one contributing cause in the operating-system comparison, which stays open.
From the labs
- L27Retraction: our benchmark laptop was running at about one fifth speed, Negative
- L28A compiler setting silently deleted our vector instructions, Negative
- L29What does an operating system cost? A pre-registered result we held back, Held back
- L30The '16% slower' lookup-table kernel claim, retracted, Negative
-
Lesson 6: A number without a measurement behind it is not a number.
Printed benchmark lines were fixed strings, a source-comment figure did not reproduce, and recipes repeated in the literature failed when we ran them. This is where our rule of provenance on every number comes from.
From the labs
-
Lesson 7: Our explanations were wrong more often than our measurements.
A mechanism we gave in four places was wrong while the counts stood, and one of our own null results was a vacuous edit. We now publish the correction next to the measurement.
From the labs
-
Lesson 8: A convenience feature can hide the failure a check exists to find.
The self-check's automatic re-download would likely mask the corruption a monitor is meant to report. This was inferred, not run on corrupted weights.
From the labs
-
Lesson 9: Integrity has a cost; report it next to the claim.
2.51 times the plain decode on one old laptop, which the source calls an upper bound, from single runs.
From the labs
-
Lesson 10: Passing a test does not mean every rule is tested.
12 of 16 single-rule breaks were caught. The tie-break and the denormal-flush pin are protected only by inputs the vector never contains.
From the labs
-
Lesson 11: A bar set in advance keeps us honest, and we say when it was not set in advance.
A 2 ULP bar was missed by the first fix and we kept the bar. A surprising result was held back by a rule we wrote in advance. The final sin/cos mechanism was added after a miss and was not itself pre-registered.
From the labs
-
Lesson 12: Everything so far has been run by us.
No unaffiliated party has reproduced the results. Until one does, every claim on this site is 'reproduced by us'.
From the labs