Training checkpoints across CPUs and instruction sets

01Why it matters

If the same training job gives different checkpoint bytes on different computers, a trained model cannot be reproduced from its recipe.

02Method, step by step

1. Train a tiny language model with a fixed seed and corpus; hash the raw parameter bytes at steps 50, 100, 150 and 200.
2. Run default PyTorch on an AVX2 CPU and a no-AVX2 CPU.
3. Run PyTorch again with the CPU kernel and thread count pinned.
4. Run a from-scratch fixed-order C trainer (13,088 parameters) on five hosts.

03Results, exact and sourced

1. R1Default PyTorch: the printed losses were identical, but every checkpoint hash differed between the two CPUs from step 50. Same-machine reruns matched on each host. The source calls the cause (different vectorized kernel path) 'consistent with', not demonstrated.

   **MISMATCH.** The four printed loss scalars are bit-identical (same Python `repr()` at every checkpoint on both hosts), but all four checkpoint sha256 digests differ, starting at the first recorded checkpoint (step 50).

2. R2Pinned PyTorch (kernel dispatch and thread count pinned) gave byte-identical checkpoints on the same two x86_64 machines; the source says this is not a cross-ISA result.

   **MATCH — cross-machine, not cross-ISA.**

3. R3The fixed-order C trainer gave identical checkpoint hashes on all five hosts, including one aarch64 phone; a second clean build on one x86_64 host and the phone reproduced the same digests.

   All four checkpoint sha256 digests are identical across all five hosts: three x86_64 machines without AVX2, one x86_64 machine with AVX2/FMA, and one aarch64 machine — a cross-ISA result.

05Scope and caveats

• Toy scale: the C trainer has 13,088 fp32 parameters and runs 200 steps; no claim about large models or step counts.
• The C trainer is a different architecture (byte-level MLP) from the PyTorch runs; no numeric parity between them is claimed.
• CPU only; no GPU host was tested.
• The C trainer source is not released.
• One run per configuration; the cross-ISA result rests on a single aarch64 device.
• The source pins no general guarantee: each result is an observed outcome for its specific configuration and host set.
• No timing figures are reported (identity/correctness experiment only).
• Our own hosts only. Run by us; no unaffiliated reproduction.

06Outcome

07What we learned

Identical printed loss numbers hid different checkpoint bytes; only hashing the raw bytes revealed it. Pinning the kernel made two x86_64 machines agree, and a fixed-order trainer gave identical bytes across instruction sets, shown so far only at toy scale on our own hosts.

08Check it yourself

• cis2-spec / 2026-09-24-CROSS-ISA-TRAINING-RECEIPT.mdhttps://github.com/Aefinity-AI/cis2-spec/blob/main/docs/results/2026-09-24-CROSS-ISA-TRAINING-RECEIPT.md

Read the result page. The C trainer source is not released.