Aefinity AI

L08 Mixed Q4 · Agent records and their cost

Receipt-verified agent evaluation, and the hole we found in our own verifier

QuestionIf every test item needs a verified receipt, can someone pass off a genuine receipt under the wrong item?

Run by us; no unaffiliated reproduction

01Why it matters

If a test score depends on receipts, someone could present a real receipt for the wrong test item. We tried to do exactly that to our own verifier.

02Method, step by step

  1. Run 60 single-tool-call items; each emits a receipt that is replayed against pinned artifacts before it is scored.
  2. Apply four hand-made tampers: flip an output token, change a tool-argument byte, truncate the chain by one step, file a valid receipt under another item.
  3. Check version 1 (no item binding) and version 2 (item id, prompt and a session nonce folded into the start of the chain).

Method as recorded

60-item single-tool-call suite (template T1), pruned BitNet-2B, one host (one of our own machines). Every item emits an agent_trace receipt that is replayed against pinned model/embed/vocab artifacts before the item is scored. One hand-made tamper was applied to each of four items: flip an output token, change a tool-arg byte, truncate the chain by one step, and replay a valid receipt under a different item's filename. v1 receipts have no context binding. v2 folds item-ctx = sha256(item_id || prompt_bytes || session_nonce) into the trace genesis. Run by us.

03Results, exact and sourced

  1. R1All 60 receipts verified.

    | Receipt verification | 60/60 PASS |

    Verbatim source cis2-spec / 2026-09-13-RECEIPT-VERIFIED-EVAL60.md

  2. R2Task score on receipt-verified outputs.

    | Score (receipt-verified outputs, arg_match AND output_match) | 45/60 |

    Verbatim source cis2-spec / 2026-09-13-RECEIPT-VERIFIED-EVAL60.md

  3. R3v1 caught 3 of 4 tampers; after context binding, v2 caught all 4.

    Overall: v1 = 57/60 PASS, **3/4 tampers caught**. v2 = 56/60 PASS, **4/4 tampers caught**.

    Verbatim source cis2-spec / 2026-09-13-RECEIPT-VERIFIED-EVAL60.md

  4. R4The v1 miss: a valid receipt filed under another item's name (lookup_hit_01) verified like a genuine one.

    **PASS (not caught)**

    Verbatim source cis2-spec / 2026-09-13-RECEIPT-VERIFIED-EVAL60.md

04What it does NOT show

Out-of-sample protection: the fix was tested on the same four cases that exposed the hole. One host, one model and one template. The 45 of 60 is a task score, not a benchmark.

05Scope and caveats

  • Four hand-made tampers, not adversarially optimized or randomly sampled; the v2 fix was tested on the same four cases that exposed the hole, so 4/4 is not out-of-sample.
  • Template T1 only (calculator, lookup, mixed single tool-call turns); one pruned BitNet-2B model; a single host (one of our own machines) generated all receipts, so there is no cross-machine claim.
  • The 45/60 score is a task score on receipt-verified outputs and is not a benchmark. The source reports no timing figures.
  • The fix lives on branch cm/safe2c-ctx-binding (commit 13ed3c47ae5a), and the source says it was not yet merged into alice-aegis main.
  • Run by us; no unaffiliated reproduction.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

A receipt can verify and still be filed on the wrong item. We found that by trying to break our own verifier, and binding each receipt to its item closed the case we found.

08Check it yourself

Read the result page and the fix commit. The source says the fix was on a branch not yet merged into main.

  • L09
    Positive

    Ordinary logging versus replay-based receipts on one-field edits

    A log is text anyone could have typed. Replaying the computation catches the mismatch without knowing in advance which field was changed.

    Q4

  • L10
    Mixed

    What receipts cost on a real agent episode

    Receipts are not free. In this episode the receipt loop took 2.51x the plain decode, and the source calls that an upper bound. On old hardware, verification took 40 s on the i5-5200U machine and minutes on the Celeron N4020. The cost should be reported next to every integrity claim. These are single runs, not a benchmark.

    Q4

  • L07
    Positive

    Realistic edits to the embedding table, including one hidden row

    Output checks are only as good as the prompts you test with. One edited row that no prompt read was missed by text replay and by the top-5 check in 40 of 40 trials, while a check over the computation caught all 40. Even when prompts did read the edited row, those output checks caught only 11 and 21 of 40.

    Q3