Aefinity AI

L18 Positive Q6 · Do our tests test anything

E28, E36, E37: every internal activation compared against an independent implementation

QuestionCould errors inside the model cancel out, so the final answer looks right while the inside is wrong?

Run by us; no unaffiliated reproduction

01Why it matters

Errors inside a model could cancel, so the final answer looks right while the middle is wrong.

02Method, step by step

  1. Feed the same token ids to our verifier and to a Hugging Face transformers fp32 forward pass.
  2. Compare every intermediate tensor.
  3. Start with one prompt, then widen to six cells on each model.

Method as recorded

Teacher-forced comparison of every intermediate tensor of the verifier against a Hugging Face transformers fp32 forward pass: one prompt on SmolLM2-135M and Qwen2.5-0.5B (E28), then six prompt/length cells on SmolLM2-135M (E36) and six on Qwen2.5-0.5B (E37). Run by us.

03Results, exact and sourced

  1. R1E28: 13,452 intermediate tensors (one prompt, 19 positions, two models: SmolLM2-135M and Qwen2.5-0.5B) compared; worst relative L2 disagreement 9.9e-5, on one tensor of one model (Qwen). This is a single-prompt maximum, not a bound for the model.

    Worst disagreement: 9.9e-5 relative

    Verbatim source cis2-spec / E28_LAYER_ORACLE.md

  2. R2E36 corrected our own published worst case: over six SmolLM2-135M prompt cells (80,565 tensors) the worst was 3.406e-05, 1.53 times the 2.228e-05 E28 published from its single SmolLM2 prompt.

    worst case is 3.406e-05, 1.53x the

    Verbatim source cis2-spec / E36_ORACLE_PROMPT_SWEEP.md

  3. R3E37: six Qwen2.5-0.5B cells add 64,260 tensors; with E36 that is 144,825 tensors across two architectures. On Qwen, E28's 9.923e-05 turned out to be the six-cell maximum (rel_rms rose to 2.883e-03); the earlier layer-ratio bounds were per-prompt and were corrected (erratum E-9).

    Taken with E36 this is 144,825 tensors across two

    Verbatim source cis2-spec / E37_QWEN_ORACLE_SWEEP.md

04What it does NOT show

Bit-exactness (the oracle is a different implementation), tokenization, or all prompts.

05Scope and caveats

  • Agreement is to about 1e-5 relative (up to ~1e-4 on Qwen), not bit-exact, because the oracle is a different implementation and must not be bit-exact; this is an accuracy check, not a determinism check.
  • Two small models; six prompts per model is still not all prompts (E37 states this).
  • The Qwen runs use supplied token ids from the oracle tokenizer and are not conformance runs; they attest to spec sections 4-11 only, not tokenization (section 3).
  • The oracle (transformers) is an independent implementation, not ground truth.
  • Not run in CI; reproducible on demand with the model artifacts and torch.
  • Run by us; no unaffiliated reproduction.

06Outcome

Positive

The check worked as expected, within the stated scope.

07What we learned

A maximum measured on one input is only that input's maximum. Widening the sweep corrected our own published number on SmolLM2 (1.53x higher), and on Qwen showed the single-prompt figure happened to be the worst case, so it was unknown, not reliably low.

08Check it yourself

Read the three pages (links above). The comparison is reproducible on demand with the model artifacts and torch; it is not run in CI.

  • L05
    Mixed

    Sin/cos accuracy for RoPE against a pre-registered 2 ULP bar (v0.2, v0.3, v0.3b)

    Fixing only the reduction improved the error a lot and still failed the bar, because the polynomial was now the dominant error. We kept the bar and changed both the reduction and the polynomials, which met it in the local harness.

    Q6

  • L13
    Mixed

    E22: which spec rules does the test vector actually exercise?

    Passing the published test does not prove every rule matters. 12 of 16 single-rule breaks were caught. The tie-break and the denormal-flush pin are protected only by inputs this vector never contains. One of our own null results turned out to be a vacuous edit that tested nothing, and we corrected it in place. Correcting it exposed a weak spot in the spec's own rsqrt check, which tests only a value where every implementation agrees.

    Q6

  • L15
    Positive

    E25: the RMSNorm multiply order matters, and our spec sentence saying otherwise was false

    A reordering that leaves the generated text unchanged (on this short vector) still changes the bits, so comparing text is not enough. A published limitation was out of date, and we corrected it in the open.

    Q6