Aefinity AI

Our test standard

A standard we state, and its limits.

This page describes the protocol as our public repository documents it. It is stated and applied by us and has not been audited by an outside party.

The practices

  1. 01

    Pre-registration: a pass mark is written before the run, then a verdict is recorded.

  2. 02

    A public, salted, hash-chained fingerprint ledger of the private register, with a verify_ledger.py script and an OpenTimestamps proof file (LEDGER.tsv.ots).

  3. 03

    No timing or throughput numbers in the spec repository.

  4. 04

    Provenance on every number.

  5. 05

    Defects recorded as numbered errata in CHANGELOG.md; pinned digests do not move silently.

  6. 06

    Mismatches are welcomed as results.

In the repository's own words

Quoted verbatim from the public cis2-spec repository.

  • At the time of this export the ledger holds 233 lines. They cover 84 registered experiments, and 65 of them have a verdict line. 161 of the 233 lines are marked backfill.

    Verbatim source cis2-spec / README.md

  • It cannot show when an experiment ran. A lab could run an experiment in private, look at the result, and then publish a pass mark.

    Verbatim source cis2-spec / README.md

  • That rule is a promise. This page cannot check it.

    Verbatim source cis2-spec / README.md

  • A number with no provenance gets removed rather than trusted.

    Verbatim source cis2-spec / CONTRIBUTING.md

  • The rule is that goldens do not move silently: if an erratum changes a pinned digest, the change, the cause and the new value all land in the same commit as the version bump.

    Verbatim source cis2-spec / CONTRIBUTING.md

  • No unqualified superlatives.

    Verbatim source cis2-spec / CONTRIBUTING.md

  • Open the PR whether your digests match or not. A mismatch is a result.

    Verbatim source cis2-spec / CONTRIBUTING.md

What the standard cannot prove

  • Stated and applied by us; no unaffiliated audit of whether practice always matched the statement.
  • 161 of 233 ledger lines are backfill and prove existence only from their publication date; backfill lines show no order of writing.
  • The ledger shows counts and line kinds, not what was tested or found.
  • The ledger count is a snapshot and will grow.
  • The ledger cannot show when an experiment ran; the rule that a run starts only after its bar line is public applies from the day of publication on, with exceptions marked start-unanchored.
  • The ledger chain verification was not re-run by this verifier; the earlier claim of an unauthenticated-clone verification is removed from the method text.

Details

Our test standard and the public experiment ledger; the standing challenge: Open challenge: the Hall of Divergence.