All labs
Showing 33 of 33 labs
-
L01Positive
CIS-2: the same AI output, bit for bit, on different machines
With the arithmetic pinned in writing, one fp32 digest reproduced across AVX2 and scalar x86, an aarch64 phone and an AVX-512 cloud CPU, in our own runs. Until an outsider runs it, the claim stays 'reproduced by us'.
Q1
-
L02Positive
An NVIDIA GPU reproduces the CPU reference digest (Tesla P100)
Our own early GPU runs passed for the wrong reason. A re-run caught it, and the claim now rests only on the corrected run.
Q2
-
L03Positive
Batching and precision on a Tesla T4: which shortcuts change the digest
In our fp32 test, the pinned order of operations decided the digest. The same order matched on CPU and GPU, and a different order broke it, which is what a digest is supposed to detect. This is one matmul on one GPU type, so it shows the check works there, not that it holds everywhere.
Q2
-
L04Mixed
Clean-room reimplementations: one mismatch (from a memo), one match (from spec v0.1), one partial that exposed a spec error
The failures taught more than the match. The C attempt showed that the spec's own prose on witness-chain order contradicted the reference. A spec-only implementer reproduced it faithfully, and the mismatch exposed the defect. Fixing the spec led to a full C match.
Q5
-
L05Mixed
Sin/cos accuracy for RoPE against a pre-registered 2 ULP bar (v0.2, v0.3, v0.3b)
Fixing only the reduction improved the error a lot and still failed the bar, because the polynomial was now the dominant error. We kept the bar and changed both the reduction and the polynomials, which met it in the local harness.
Q6
-
L06Positive
One flipped bit in the weights: which checks notice?
In the larger run, among the 247 trials where the output actually changed, the text-replay column missed 182 and the top-k/receipt-style check missed 82. Some flips (43) changed nothing in the computation. A check on the computation itself is by definition silent only on flips that changed nothing, but that 0/247 is true by construction, so the informative finding is how often the weaker checks miss. Single prompt and model, harness-chosen flips, no adversary.
Q3
-
L07Positive
Realistic edits to the embedding table, including one hidden row
Output checks are only as good as the prompts you test with. One edited row that no prompt read was missed by text replay and by the top-5 check in 40 of 40 trials, while a check over the computation caught all 40. Even when prompts did read the edited row, those output checks caught only 11 and 21 of 40.
Q3
-
L08Mixed
Receipt-verified agent evaluation, and the hole we found in our own verifier
A receipt can verify and still be filed on the wrong item. We found that by trying to break our own verifier, and binding each receipt to its item closed the case we found.
Q4
-
L09Positive
Ordinary logging versus replay-based receipts on one-field edits
A log is text anyone could have typed. Replaying the computation catches the mismatch without knowing in advance which field was changed.
Q4
-
L10Mixed
What receipts cost on a real agent episode
Receipts are not free. In this episode the receipt loop took 2.51x the plain decode, and the source calls that an upper bound. On old hardware, verification took 40 s on the i5-5200U machine and minutes on the Celeron N4020. The cost should be reported next to every integrity claim. These are single runs, not a benchmark.
Q4
-
L11Mixed
Continuous validation: does a one-byte corruption get caught, and what hides it?
The hash gate catches a one-byte change. But a convenience feature, the automatic re-download, would likely hide exactly the corruption a monitor exists to report (inferred, not run on corrupted input).
Q3
-
L12Mixed
Training checkpoints across CPUs and instruction sets
Identical printed loss numbers hid different checkpoint bytes; only hashing the raw bytes revealed it. Pinning the kernel made two x86_64 machines agree, and a fixed-order trainer gave identical bytes across instruction sets, shown so far only at toy scale on our own hosts.
Q1
-
L13Mixed
E22: which spec rules does the test vector actually exercise?
Passing the published test does not prove every rule matters. 12 of 16 single-rule breaks were caught. The tie-break and the denormal-flush pin are protected only by inputs this vector never contains. One of our own null results turned out to be a vacuous edit that tested nothing, and we corrected it in place. Correcting it exposed a weak spot in the spec's own rsqrt check, which tests only a value where every implementation agrees.
Q6
-
L14Mixed
E24: the spec never says which Unicode version defines a letter
A spec can be exact about arithmetic and still leave the input text under-specified. Claims like 'generated from the Unicode database' need a check pinned to the environment.
Q5
-
L15Positive
E25: the RMSNorm multiply order matters, and our spec sentence saying otherwise was false
A reordering that leaves the generated text unchanged (on this short vector) still changes the bits, so comparing text is not enough. A published limitation was out of date, and we corrected it in the open.
Q6
-
L16Mixed
E26: sin/cos at very long contexts — the sampled answer was wrong
4,096 samples per binade never hit the worst points among about 8.4 million, so sampling gave a comfortable answer that was wrong. An exhaustive sweep replaced it. Absolute error stays flat while ULP error grows near zeros of sin/cos, so ULP alone is the wrong figure of merit for these functions.
Q6
-
L17Mixed
E27: exp, ln and SiLU on every float32 input, and a defect in our spec
The accuracy was the unremarkable part. Exhaustive testing found a real defect in our own spec that sampled tests could not, and we recorded it in the open.
Q6
-
L18Positive
E28, E36, E37: every internal activation compared against an independent implementation
A maximum measured on one input is only that input's maximum. Widening the sweep corrected our own published number on SmolLM2 (1.53x higher), and on Qwen showed the single-prompt figure happened to be the worst case, so it was unknown, not reliably low.
Q6
-
L19Undated in the sourcePositive
E29: does the compiler's optimiser change the bits?
Optimisation was not the enemy for this decode. The pinned order of operations is preserved in the emitted code, so builds with very different vector instruction counts produced identical bytes. This is measured on one compiler version and on the tested models and prompts.
Q1
-
L20Null
E35 to E40: does the flush-to-zero rule ever matter? (after two wrong explanations)
We never found a case where the flush rule changes the digest. The explanations we first gave for why were wrong, and we published the corrections next to the exhaustive count that replaced them.
Q6
-
L21Negative
E41: the spec pins how text becomes tokens, but not how tokens become text
Matching digests is not the same as matching meaning. The missing piece is a text-level definition; whether v0.4 specifies detokenization or declares it out of scope is a spec-owner decision.
Q5
-
L22Mixed
ev-1: a 160-item mini evaluation on the pinned 135M model
The model scores poorly but repeats exactly on one machine. We report the poor accuracy alongside the perfect same-host repeatability. Cross-host verification is still open.
Q10
-
L23Mixed
Our test standard and the public experiment ledger
We publish how we test, including what the method cannot prove, and we correct our own published claims through numbered errata.
Q7
-
L24Null
Open challenge: the Hall of Divergence
As of 2026-10-12 no one outside the lab has reproduced or broken the CIS-2 results, and the page says so plainly. The only listed runs are our own, and their logs are not yet public.
Q7
-
L25Mixed
RANGER mock run: low-bit quantization ideas tested on synthetic tensors
On synthetic data, one sub-hypothesis was falsified and two design constraints emerged: rotate only after splitting off the sparse outliers, and rotation helps 4-bit activations but not below 4 bits. A later test showed width and rotation overlap on outliers. These are mechanism-level findings, not claims about real models.
Q9
-
L26Undated in the sourceMixed
RANGER Phase 1: the quantization ideas on real models (SmolLM2-135M, Qwen3-0.6B)
A prior null result was retracted publicly. Rotation recovery replicated across two architectures (92.3% and 92.5%), while the census-replaces-rotation shortcut was refuted, and a super-weight lever is null at 135M and only small at 0.6B. One of our own sub-results (0.38 rescue) was refuted by our confirmation pass.
Q9
-
L27Negative
Retraction: our benchmark laptop was running at about one fifth speed
Measure the machine before measuring the code. We retracted every earlier number from that box instead of reporting a speed-up.
Q8
-
L28Negative
A compiler setting silently deleted our vector instructions
Trust the binary you ship, not the annotation you wrote. Nothing failed and nothing warned; only counting instructions in the artifact exposed it. A mechanical check (assert nonzero ymm/vfmadd count in CI) is the fix, not a benchmark.
Q8
-
L29Held back
What does an operating system cost? A pre-registered result we held back
A surprising result was held back by a rule we had written in advance. We found one contributing cause in our own build (soft-float), but the question remains open; the numbers will be published when the mechanism is understood.
Q8
-
L30Negative
The '16% slower' lookup-table kernel claim, retracted
A number cited from a source file but never measured was wrong in the form claimed. The conclusion survived, for a different reason.
Q8
-
L31Negative
Audit of our own removed code: benchmark lines that measured nothing
Every published number needs a machine, a log and a source behind it. This audit is where that rule came from.
Q8
-
L32Null
E-S1: how far can a ternary 2B model be compressed without loss?
Pretrained ternary weights are close to uniform, so lossless coding gains little. Real savings would need changes to the model, not a better coder.
Q9
-
L33Undated in the sourceNegative
Claims from the literature that failed when we ran them
Several widely repeated recipe claims failed when we checked them, so we keep a public list of what failed. For each claim, the list says whether we ran it ourselves or checked it against a source's stated requirements.
Q9
No lab matches both filters.