Ten questions. What we found, and what is still open.

What we found so far

In our own runs, yes, for one narrow case: fp32 greedy decoding of one small model (SmolLM2-135M, 16 tokens), once the arithmetic is pinned in a written specification. The same digest came out on AVX2 and scalar x86 machines, an aarch64 phone and a free cloud CPU. Ten differently optimised x86_64 builds produced one identical dump. For training, default PyTorch gave different checkpoints on two CPUs even though the printed losses were identical, while a fixed-order C trainer gave identical checkpoints on five hosts including a phone, at toy scale (13,088 parameters).

Still open

Whether anyone outside our lab gets the same result; larger models; production precisions such as bf16; sampling, batching and quantisation. We have not tested those.

What we found so far

On one Tesla P100, an independent CUDA implementation matched the CPU digests, after two earlier GPU runs turned out to be false positives (the binary had really been built for the CPU) and were withdrawn. On a Tesla T4, one matrix multiply gave identical results with row-batched fp32 at batch sizes 1, 4 and 8, while grouped fp32 and int8 changed the result at every batch size, as predicted before the run.

Still open

Other GPU generations, tensor cores, whole-model batching beyond row-batched fp32, multi-GPU serving, and performance. The CUDA code is not public.

What we found so far

Re-reading the output is a weak detector. In one run of 295 bit-flip trials, among the 247 where the output changed, a replay-style column missed 182 and a receipt-style check missed 82. An edited embedding row that no test prompt read was missed 40 of 40 times by text replay and by a top-5 check, while a check over the full computation caught all 40. A one-byte file corruption was stopped by a hash gate, but our own self-check would likely have repaired the file and hidden the corruption from a monitor. Also, 43 flips changed nothing in the computation, which a computation-level check by design does not flag.

Still open

Adversaries who optimise their tampering; other prompts, models and tensors; a scheduled, multi-host monitoring deployment.

What we found so far

A receipt that is replayed against pinned artifacts can catch edits an ordinary log display does not flag. Trying to break our own verifier found a hole: a genuine receipt filed under the wrong test item verified like a real one. Binding each receipt to its item closed that case. Receipts are not free: on one old laptop the episode with receipts took 2.51 times as long as a plain decode (the source calls that an upper bound), and verification took 40 seconds there and minutes on a weaker machine.

Still open

Whether the fix holds against tampering that was not hand-picked; cost on other hardware and at scale.

What we found so far

Partly. A Rust version written from an internal memo did not match. A Rust version written from spec v0.1 matched all digests on its first CI run. A C version written from spec v0.2 matched three of four digests, and the mismatch led us to a defect in our own spec: the written order of the witness chain disagreed with the reference. We then found two more gaps by reading our spec critically: it names no Unicode version for letters, and it pins text to tokens but not tokens to text.

Still open

Independent humans doing the same; every one of these implementers was an AI coding agent run by one operator inside our organisation.

What we found so far

We broke our spec's rules one at a time: the official test noticed 12 of 16 breaks, and not the tie-break or the denormal-flush pin. Sampled accuracy checks gave comfortable wrong answers (at most 1 ULP) where exhaustive sweeps found much larger ULP errors near zeros of sin and cos. Exhaustive testing of exp found a real defect in a safety guard in our spec. A pre-registered 2 ULP bar was missed by our first fix and met by a second. We also retracted two wrong explanations of why the flush-to-zero rule never mattered.

Still open

Whether the test vector exercises every rule; the untriggered rules are protected only by separate self-tests.

What we found so far

They cannot tell completely, and our own ledger page says so. We publish pass marks before runs, a hash-chained public ledger of the private register, numbered errata and a rule that mismatches count as results. The ledger cannot show when an experiment ran. As of 2026-10-12 nobody outside the lab has reproduced or broken our results, and the public Hall of Divergence says so plainly.

Still open

An outside audit of whether our practice always matched our statement.

What we found so far

Ours did not, at first. Our benchmark laptop was clamped to about a fifth of its clock, so every earlier number from it was retracted. A compiler setting removed all vector instructions and the binary ran 217 to 258 times slower. A pre-registered comparison of a no-OS build against a minimal Linux was held back after a surprising first result; we found one contributing cause in our own build (soft-float), and the question remains open. A '16% slower' claim copied from a source comment did not reproduce. An early version of our software printed benchmark lines that were fixed strings.

Still open

A clean, controlled speed comparison across machines. We make no general speed claim; the speed figures on this site are single measurements on named machines.

What we found so far

Some, within narrow limits. On two real small models, a rotation recovered 92.3% and 92.5% of the 4-bit activation collapse, though perplexity stayed above the baseline. A shortcut that skips the rotation was refuted, a super-weight lever was null at 135M parameters, and one of our own earlier null results was retracted. Lossless coding of a ternary 2B model reached 1.532 bits per weight, about 4% below ideal packing. A stock soft-prompt recipe gave outputs wrong by about 300 times on one layer when we ran it.

Still open

Models above about 0.6B parameters, quantisation-aware training, and deployable int4 kernels.

What we found so far

The 160-item evaluation repeated exactly on one machine (20 of 20 self-consistency items). The model scored poorly, which was expected from a 135M base model with simple heuristics: 3 of 60 on truthfulness, 5 of 60 on a tool-use proxy and 10 of 20 on refusal probes, which the source says is chance. The point was repeatability, not accuracy.

Still open

Cross-host verification of the evaluation, and anything about larger or instruction-tuned models.