Aefinity AI

L21 Negative Q5 · Rebuilt from the written spec

E41: the spec pins how text becomes tokens, but not how tokens become text

QuestionCould two certified programs agree on every number in a receipt yet disagree on what it says?

Run by us; no unaffiliated reproduction

01Why it matters

A receipt is read by people. If two conforming programs can print different text for the same digests, the receipt's meaning is not pinned.

02Method, step by step

  1. Read the spec against the tokenizers of the two named checkpoints.
  2. Count vocabulary entries and output indices where turning ids back into text is ambiguous.
  3. Probe with small Rust examples.

Method as recorded

Read CIS-2 spec v0.3b against the tokenizers of the two checkpoints it names (SmolLM2-135M and Qwen2.5-0.5B) and counted where turning ids back into text is ambiguous. Probes were Rust examples run on one host (Debian 13, x86_64 VM). Run by us.

03Results, exact and sourced

  1. R1Two conforming implementations can disagree on a receipt's text.

    two conforming implementations can disagree about what a receipt says while agreeing on every digest in it.

    Verbatim source cis2-spec / E41_DETOKENIZATION_IS_UNSPECIFIED.md

  2. R2313 of SmolLM2's 49,152 vocabulary entries are not valid UTF-8 on their own.

    **313** of SmolLM2's 49,152 vocab entries are not valid UTF-8 alone (0.64 %)

    Verbatim source cis2-spec / E41_DETOKENIZATION_IS_UNSPECIFIED.md

  3. R3271 of Qwen's 151,936 output indices map to no string in the tokenizer files.

    **271** of Qwen's 151,936 logit indices are in neither `model.vocab` nor `added_tokens`

    Verbatim source cis2-spec / E41_DETOKENIZATION_IS_UNSPECIFIED.md

  4. R4All 17 of SmolLM2's special tokens decode to text that re-encodes to different ids, so a rendered control token cannot be told from text spelling it.

    **17** of 17 SmolLM2 `added_tokens` decode to text that re-encodes to *different* ids

    Verbatim source cis2-spec / E41_DETOKENIZATION_IS_UNSPECIFIED.md

04What it does NOT show

That any real receipt was misread, that other tokenizers behave the same, or that it affects conforming runs on Qwen.

05Scope and caveats

  • A gap in our spec (v0.3b), not a defect in the checkpoints; recorded as erratum E-14.
  • Two checkpoints only; the counts are properties of those artifacts, not of tokenizers in general.
  • The 271 unmapped Qwen ids did not win the argmax on one prompt over 16 decode steps (untriggered on this vector, not shown impossible); no search over prompts was run.
  • Qwen's tokenizer.json is refused by the spec's conforming path, so the Qwen gap does not affect conforming runs.
  • No survey of other inference specifications; no claim the gap is unusual.
  • Run by us on one machine; no unaffiliated reproduction.

06Outcome

Negative

The claim or approach failed, or we retracted it.

07What we learned

Matching digests is not the same as matching meaning. The missing piece is a text-level definition; whether v0.4 specifies detokenization or declares it out of scope is a spec-owner decision.

08Check it yourself

Read the page (link above).