Aefinity AI

L23 Mixed Q7 · No tuning after the fact

Our test standard and the public experiment ledger

QuestionHow can an outsider tell that a lab didn't tune its results after the fact, rewrite them later, or quietly drop the failures?

Run by us; no unaffiliated reproduction

01Why it matters

Results are only as credible as the process behind them.

02Method, step by step

  1. Write the pass mark before the run.
  2. Publish a salted, hash-chained fingerprint of the private register.
  3. Number every defect as an erratum.
  4. Remove numbers that have no provenance.

Method as recorded

Stated practices in public files: (1) pre-registration, a pass mark written before the run, then a verdict; (2) a public salted, hash-chained fingerprint ledger of the private register, with a verify_ledger.py script and an OpenTimestamps proof file (LEDGER.tsv.ots); (3) no timing or throughput numbers in the spec repo; (4) provenance on every number; (5) defects recorded as numbered errata in CHANGELOG.md, and pinned digests do not move silently; (6) mismatches are welcomed as results. The standard is stated and applied by us, not audited by an outside party. All six URLs returned HTTP 200 unauthenticated (github.com and raw.githubusercontent.com) on 2026-10-12.

03Results, exact and sourced

  1. R1The ledger covers 84 registered experiments, 65 with a verdict; 161 of 233 lines are backfill.

    At the time of this export the ledger holds 233 lines. They cover 84 registered experiments, and 65 of them have a verdict line. 161 of the 233 lines are marked backfill.

    Verbatim source cis2-spec / README.md

  2. R2The ledger states what it cannot prove.

    It cannot show when an experiment ran. A lab could run an experiment in private, look at the result, and then publish a pass mark.

    Verbatim source cis2-spec / README.md

  3. R3The start rule is a promise the page itself cannot check.

    That rule is a promise. This page cannot check it.

    Verbatim source cis2-spec / README.md

  4. R4Provenance on every number, or the number is removed.

    A number with no provenance gets removed rather than trusted.

    Verbatim source cis2-spec / CONTRIBUTING.md

  5. R5Pinned digests do not move silently.

    The rule is that goldens do not move silently: if an erratum changes a pinned digest, the change, the cause and the new value all land in the same commit as the version bump.

    Verbatim source cis2-spec / CONTRIBUTING.md

  6. R6No unqualified superlatives.

    **No unqualified superlatives.**

    Verbatim source cis2-spec / CONTRIBUTING.md

  7. R7A mismatch is a result.

    Open the PR whether your digests match or not. A mismatch is a result.

    Verbatim source cis2-spec / CONTRIBUTING.md

04What it does NOT show

That we always followed the standard; no outsider has audited it.

05Scope and caveats

  • Stated and applied by us; no unaffiliated audit of whether practice always matched the statement.
  • 161 of 233 ledger lines are backfill and prove existence only from their publication date; backfill lines show no order of writing.
  • The ledger shows counts and line kinds, not what was tested or found.
  • The ledger count is a snapshot and will grow.
  • The ledger cannot show when an experiment ran; the rule that a run starts only after its bar line is public applies from the day of publication on, with exceptions marked start-unanchored.
  • The ledger chain verification was not re-run by this verifier; the earlier claim of an unauthenticated-clone verification is removed from the method text.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

We publish how we test, including what the method cannot prove, and we correct our own published claims through numbered errata.

08Check it yourself

Read the ledger README, LEDGER.tsv and verify_ledger.py, then CONTRIBUTING.md and CHANGELOG.md (links above). We did not re-run the chain verification for this page.

  • L24
    Null

    Open challenge: the Hall of Divergence

    As of 2026-10-12 no one outside the lab has reproduced or broken the CIS-2 results, and the page says so plainly. The only listed runs are our own, and their logs are not yet public.

    Q7

  • L22
    Mixed

    ev-1: a 160-item mini evaluation on the pinned 135M model

    The model scores poorly but repeats exactly on one machine. We report the poor accuracy alongside the perfect same-host repeatability. Cross-host verification is still open.

    Q10