Aefinity AI

L14 Mixed Q5 · Rebuilt from the written spec

E24: the spec never says which Unicode version defines a letter

QuestionCan two programs that follow the spec exactly still split the same text differently?

Run by us; no unaffiliated reproduction

01Why it matters

Splitting text into tokens depends on which characters count as letters. Two programs that follow the spec could split differently.

02Method, step by step

  1. Notice our own CI check fail the same way on x86_64 and aarch64.
  2. Generate the letter, number and whitespace tables under Unicode 15.0.0 and 15.1.0 and difference the sets.
  3. Encode four test strings with both tables.

Method as recorded

A CI check that regenerates the verifier's Unicode tables (cis2-verify/tools/gen_unicode.py) failed identically on the x86_64 and aarch64 GitHub runners (run 34340397775), so it was not an ISA difference. The difference between Unicode 15.0.0 and 15.1.0 was then measured by generating the LETTER, NUMBER and WHITESPACE tables twice in one session (unicodedata2==15.0.0 vs the interpreter's own 15.1.0 database) and differencing the codepoint sets. Measured on the development box (Debian 13, x86_64, crosvm guest) and GitHub ubuntu-24.04 / ubuntu-24.04-arm runners. Run by us. Concerns spec v0.3b section 3.1.4.

03Results, exact and sourced

  1. R1The LETTER table grew by 622 codepoints between Unicode 15.0.0 and 15.1.0 (CJK Unified Ideographs Extension I). The NUMBER and WHITESPACE tables did not change.

    | LETTER | `\p{L}`, `char::is_alphabetic` | 136,104 cp | 136,726 cp | **+622: U+2EBF0..U+2EE5D** |

    Verbatim source cis2-spec / E24_UNICODE_VERSION_GAP.md

  2. R2Spec v0.3b names no Unicode version for the pre-tokenizer's property classes.

    Neither the clause nor any other part of the document names a Unicode version.

    Verbatim source cis2-spec / E24_UNICODE_VERSION_GAP.md

  3. R3Bounded: this is a finding, not a reproduction failure, and no published digest moves.

    **Status:** finding, bounded. Not a reproduction failure. No published digest moves.

    Verbatim source cis2-spec / E24_UNICODE_VERSION_GAP.md

04What it does NOT show

Any change to the published digests; the finding is bounded. The spec text itself is unchanged and the fix is proposed for v0.4.

05Scope and caveats

  • Found by our own CI, not an outside report.
  • Run by us; no unaffiliated reproduction.
  • Harmless for the pinned artifacts: for the pinned SmolLM2-135M vocabulary, four test strings encoded to identical token ids with the 15.0.0 and 15.1.0 tables. The source says this holds by property of the vocabulary, not by anything the spec states, and a future Unicode release could diverge.
  • The spec amendment pinning Unicode 15.1.0 is only proposed for v0.4; the fix is a maintainer decision.
  • Only the checkability of the generated file was fixed (REQUIRED_UCD pin and CI install of unicodedata2==15.1.0); the spec text itself is unchanged.
  • The source states no timing number from these runs may be used.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

A spec can be exact about arithmetic and still leave the input text under-specified. Claims like 'generated from the Unicode database' need a check pinned to the environment.

08Check it yourself

Read the finding (link above).