Aefinity AI

L13 Mixed Q6 · Do our tests test anything

E22: which spec rules does the test vector actually exercise?

QuestionIf we deliberately break one rule of our own spec, does the official test notice?

Run by us; no unaffiliated reproduction

01Why it matters

A passing test only means something if breaking the rules makes it fail. We broke our own rules to see how much the test notices.

02Method, step by step

  1. Make 16 single-rule mutations of the C verifier.
  2. Build and run each against the pinned vector on one host.
  3. Include unmutated builds at four optimisation levels as controls.
  4. Count a rule as exercised if breaking it moves the digest.

Method as recorded

Sixteen single-clause mutations of the C verifier (CIS-2 v0.3b), each built and run against the pinned test vector (a single 4-token prompt on a 135M checkpoint with head_dim = 64) on one host (an Intel i5-5200U machine, gcc 14.2.0), plus unmutated builds at four optimisation levels as positive controls. A clause counts as exercised if breaking it moves the witness digest. Run by us; one machine, one compiler.

03Results, exact and sourced

16 single-rule mutations of the spec: did the digest move?
  • 12 Digest moved
  • 3 Digest unchanged
  • 1 Declared skipped

Digest moved: 12; Digest unchanged: 3; Declared skipped: 1

Each square is one mutation. Unchanged rows include the tie-break, which the vector never reaches, and a vacuous mutation the authors corrected. Figure drawn from the sourced results below.

  1. R112 of 16 mutations moved the digest, 3 did not, 1 was declared skipped (M07, skipped by the leg).

    **12 of 16 mutations moved the digest. 3 did not. 1 was declared skipped.**

    Verbatim source cis2-spec / E22_NECESSITY_MATRIX.md

  2. R2The argmax tie-break rule (M15, section 11) is not exercised by this vector because no tie occurs in any of its 16 argmaxes over a 49152-wide fp32 logit vector. The source says the clause is correct and necessary in general and that the vector never reaches the branch.

    **M15** | **§11 argmax tie-break** | **SAME** | **not exercised**

    Verbatim source cis2-spec / E22_NECESSITY_MATRIX.md

  3. R3Correction by the authors: the M16 row (attention scale via pinned rsqrt) was a vacuous mutation, because it substituted the function for its own body, so it tested nothing. The first published explanation (head_dim = 64 is a perfect square) was withdrawn as a rationalisation.

    **VACUOUS MUTATION — see correction below**

    Verbatim source cis2-spec / E22_NECESSITY_MATRIX.md

  4. R4A rule that can be broken without effect on the digest is untested by this vector, not redundant. This is a statement about the conformance suite, not the spec.

    A clause that can be broken with no observable effect is not thereby redundant

    Verbatim source cis2-spec / E22_NECESSITY_MATRIX.md

  5. R5Unmutated builds reproduced the baseline digest at four optimisation levels (O0, O1, O3, Os). Scope is this compiler on this host only.

    -O0 / -O1 / -O3 / -Os → d82743059d… SAME

    Verbatim source cis2-spec / E22_NECESSITY_MATRIX.md

04What it does NOT show

That a rule is redundant just because the vector does not exercise it. Also not shown: any result on another compiler or machine.

05Scope and caveats

  • One compiler (gcc 14.2.0), one machine (i5-5200U), one 4-token prompt on one 135M checkpoint with head_dim = 64. The source says the silence is a property of this vector, not of the spec.
  • Run by us; no unaffiliated reproduction.
  • M01 (section 1.3 FTZ/DAZ pin) was revised after the first run by E38, E39 (erratum E-12) and E40 (erratum E-13). Its null is now explained: no denormal arises anywhere in the decode on the two checkpoints tested (E40 counted 562,531,070,920 intermediates, 0 subnormal). The clause is untriggered here, not unnecessary, and there is still no end-to-end necessity witness. The spec guards it separately: section 15 item 4 requires its self-test to pass independently of the digest.
  • M07 (section 6.1) was skipped, so it is neither a hit nor a miss.
  • The M16 correction and rsqrt route table were measured on a different host (a ChromeOS Crostini VM, gcc -O2) than the main matrix. The source says the section 6.1 check rsqrt(64.0) == 0.125 cannot discriminate between implementations. Op-level goldens for rsqrt and the tie-break were added later in cis2-verify (follow-up items 2 and 3 marked done).
  • The source reports no timing figures from this run.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

Passing the published test does not prove every rule matters. 12 of 16 single-rule breaks were caught. The tie-break and the denormal-flush pin are protected only by inputs this vector never contains. One of our own null results turned out to be a vacuous edit that tested nothing, and we corrected it in place. Correcting it exposed a weak spot in the spec's own rsqrt check, which tests only a value where every implementation agrees.

08Check it yourself

Read the matrix page and the corrections inside it (link above).