Realistic edits to the embedding table, including one hidden row

01Why it matters

A realistic model edit may touch only rare words. If your tests never use those words, output checks pass while the model has changed.

02Method, step by step

1. Make realistic edits to the embedding table of one pruned BitNet-2B checkpoint: INT8 round-trip, 1% pruning, an approximated fine-tune patch, and single-row patches.
2. For single rows, pick 8 rows the prompts never read and 8 rows they do read, with 5 prompts each (40 trials per case).
3. Compare three detectors against the unedited baseline: text replay, top-5 logit set, witness digest.
4. Report counts with Wilson 95% intervals. A same-shape checkpoint substitution was attempted in design and found infeasible.

03Results, exact and sourced

1. R1For a single edited embedding row that none of the test prompts read (8 random rows x 5 prompts = 40 trials), text replay and the top-5 check each detected 0 of 40; the witness digest detected 40 of 40.

   When the row is not exercised by the test prompts, A and B miss it completely (0/40 each in (b')).

2. R2Control with 8 rows the prompts do read (40 trials): the top-5 check detected 21 of 40 (52.5%, Wilson 37.5-67.1%); text replay detected 11 of 40 (27.5%). Even when rows are read, detection is only partial.

   | B (top-5 logit-index check) | 21/40 | 52.5% | 37.5–67.1% | 0/40 (0.0%) |

3. R31% magnitude pruning of the whole table (20 prompts): text replay detected 18 of 20 (90.0%, Wilson 69.9-97.2%); the top-5 check and witness digest detected 20 of 20.

   | A (text replay) | 18/20 | 90.0% | 69.9–97.2% |

4. R4In the classes that were run, the witness digest detected every trial. Here 'detected' means the digest differs from baseline; the document reports no false-positive test for it.

   The witness digest (C) is the only detector of the three that caught every trial in every class tested here

5. R5UNABLE: substituting a different same-shape checkpoint was found infeasible and not run (no independently trained checkpoint with matching shape was available).

   ### (c) Different same-shape checkpoint — found infeasible

05Scope and caveats

• Small samples (n=20 or n=40), wide intervals; one checkpoint, one prompt set, embedding table only.
• The single-row classes used only a 5-prompt subset of the 20 prompts, and 8 rows; the 40 trials are 8 rows x 5 prompts, not 40 independent rows.
• The edit is +1 step of the int8-quantized code on all 2,560 elements of the row, not a bf16 last-bit edit.
• The INT8 class changed 96.91% of values (INT8 round-trip: all three detectors 20/20); the document says this does not show A/B would catch a smaller quantization.
• The fine-tune class is a labelled approximation (token-bigram nudge), not a real gradient step; the document calls its all-caught result weaker evidence than it looks.
• Row choice was a fixed-seed PRNG; the control (b'') drew rows from the prompts' own tokens, a looser selection than (b'). An attacker who knows the test prompts, or a patch confined to rare tokens, was not tested.
• The harness is not published, so the experiment cannot be rerun from the repo.
• Run by us; no unaffiliated reproduction.

06Outcome

07What we learned

Output checks are only as good as the prompts you test with. One edited row that no prompt read was missed by text replay and by the top-5 check in 40 of 40 trials, while a check over the computation caught all 40. Even when prompts did read the edited row, those output checks caught only 11 and 21 of 40.

08Check it yourself

• cis2-spec / 2026-09-17-EMBED-SUBSTITUTION-HEADTOHEAD.mdhttps://github.com/Aefinity-AI/cis2-spec/blob/main/docs/results/2026-09-17-EMBED-SUBSTITUTION-HEADTOHEAD.md

Read the result page. The harness is not yet in a public repo, so the experiment cannot be rerun from public files.