Aefinity AI

L04 Mixed Q5 · Rebuilt from the written spec

Clean-room reimplementations: one mismatch (from a memo), one match (from spec v0.1), one partial that exposed a spec error

QuestionCan someone who has never seen our code build the same bits from the written description alone?

Run by us; no unaffiliated reproduction

01Why it matters

A specification is only useful if someone who never saw the original code can build the same thing from it.

02Method, step by step

  1. Give isolated AI coding agents the written material only, never the reference source.
  2. Attempt 1 (Rust) worked from an internal memo; attempt 2 (Rust) from spec v0.1; attempt 3 (C11) from spec v0.2.
  3. Compare their four digests and 16 token ids with the reference on x86_64 and aarch64, with gcc and clang.
  4. When the C version missed one digest, dump a hash per tensor from both and find where they first differ.

Method as recorded

Isolated AI coding agents run by one operator wrote verifiers without opening the reference source. E15e (Rust) worked from an internal design memo plus two result docs, not the spec. E15g (Rust) worked from spec v0.1 only. E15j (C11) worked from spec v0.2 plus the E15d digests doc, on x86_64 and aarch64 with gcc and clang. E15k localised the C mismatch by dumping a SHA-256 digest per tensor for both implementations, then inspected the witness-chain code. Run by us. Every cited CI run is private. All four pages are public and return 200.

03Results, exact and sourced

  1. R1First Rust attempt (E15e, built from a design memo, not the spec) did not match: inv_freq digest and the full witness digest differed. The token ids and argmax digest matched.

    ## Verdict: MISMATCH, but an informative one

    Verbatim source cis2-spec / E15e_CLEANROOM_RESULT.md

  2. R2Second Rust attempt (E15g, spec v0.1 only) matched all four digests and all 16 token ids on x86_64 and aarch64 on its first CI run. The CI run is private.

    **Status: MATCH. Full bit-for-bit conformance on first CI run, no fixes needed.**

    Verbatim source cis2-spec / E15g_CLEANROOM_V2_RESULT.md

  3. R3The E15g implementer read only the v0.1 spec text for algorithm content. It also read build and CI scaffolding files, which are logged.

    implemented reading only

    Verbatim source cis2-spec / E15g_CLEANROOM_V2_RESULT.md

  4. R4C11 attempt (E15j, spec v0.2) matched 3 of 4 digests on all four cells (x86_64 and aarch64, gcc and clang). Only the witness digest CIS2_REF differed. The 16 token ids matched.

    Result: PARTIAL conformance — 3 of 4 digests match bit-for-bit, cross-ISA/cross-compiler-identical

    Verbatim source cis2-spec / E15j_CLEANROOM_C_RESULT.md

  5. R5The cause was the order of the five items in the witness chain, not the forward pass. Per-tensor digests (about 124 per run, decode steps 0 and 1 only) matched.

    root cause found — witness-chain item ORDER, not the forward pass

    Verbatim source cis2-spec / E15k_DIVERGENCE_LOCALIZATION.md

  6. R6The C code had copied the item order from the spec's written prose, which disagreed with the reference. This is classed as a spec defect.

    copied verbatim from `docs/CIS2_SPEC_v0.2.md` §12.1's written prose

    Verbatim source cis2-spec / E15k_DIVERGENCE_LOCALIZATION.md

  7. R7After we corrected the spec and the C code, all four C cells reproduced the pinned CIS2_REF. We corrected the spec ourselves after seeing the mismatch, so this is a post-fix match.

    E15j is now a full 4/4-digest, cross-language (Rust vs from-spec-text-only C11), cross-ISA (x86_64/aarch64), cross-compiler (gcc/clang) conformance PASS

    Verbatim source cis2-spec / E15k_DIVERGENCE_LOCALIZATION.md

04What it does NOT show

That independent humans can do it; the implementers were AI agents run by one operator inside our organisation. The cited CI runs are private.

05Scope and caveats

  • Implementers were AI coding agents run by one operator inside our organisation, not independent humans. No unaffiliated reproduction.
  • E15e was built from an internal design memo and result docs, not the published spec, so it is not a spec-text-only test.
  • E15j also read the E15d digests doc, so it was not strictly spec-text-only. E15g read build and CI scaffolding files, logged as not algorithmic.
  • The C match in E15k came only after we fixed the spec and the C code. The spec-as-written produced only a partial match.
  • Targets were spec v0.1 and v0.2, since superseded by v0.3b.
  • All cited CI runs are private and cannot be checked externally. The result docs are our own historical development notes.
  • Model was SmolLM2-135M, fp32, one prompt (4 prompt tokens, 16 generated tokens). Sample size is one prompt per attempt.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

The failures taught more than the match. The C attempt showed that the spec's own prose on witness-chain order contradicted the reference. A spec-only implementer reproduced it faithfully, and the mismatch exposed the defect. Fixing the spec led to a full C match.

08Check it yourself

Read the four result pages (links above). The CI runs they cite are private and cannot be checked externally.