Aefinity AI

L33 Undated in the source Negative Q9 · Shrinking models

Claims from the literature that failed when we ran them

QuestionDo published recipes for training small ternary models work as claimed on ordinary hardware?

Run by us; no unaffiliated reproduction

01Why it matters

Recipes repeated in papers and blog posts can fail on ordinary hardware.

02Method, step by step

  1. Record each cited recipe as refuted, corrected or surviving.
  2. Run it on our target box where we can.
  3. Where we cannot, say so and cite the requirement.

Method as recorded

Our research notes record each cited recipe as refuted, corrected or surviving. The soft-prompt recipe was refuted by execution on our own target machine. The T4 claim was refuted by reading the requirements stated in a cited source, and we did not run it on a T4. Run by us.

03Results, exact and sourced

  1. R1A stock Hugging Face transformers recipe (AutoBitLinear, online_quant=False) for training soft prompts on the Falcon-E checkpoint gave outputs wrong by about 300 times on one real layer (layer-0 q_proj), as measured by us on our target box.

    so the stock recipe's outputs are wrong by weight_scale^2 (~301.9x measured on real layer-0 q_proj)

    Verbatim source alice-aegis / refuted_claims.md

  2. R2The claim that stock onebitllms runs on a free Kaggle/Colab T4 in 2-5 hours is marked REFUTED in our notes. The basis is a documented hardware requirement (Triton >=3.3 needs compute capability 8.0+, and the T4 is sm_75). We did not run it on a T4. The notes say the path may survive only in an unverified older-library environment.

    Triton >=3.3 (bundled with torch >=2.7) requires NVIDIA CC 8.0+ and dropped Turing

    Verbatim source alice-aegis / refuted_claims.md

04What it does NOT show

That the recipes fail everywhere. This is our environment and one set of library versions.

05Scope and caveats

  • Our environment and one set of library versions.
  • Entries are our own research notes made public.
  • Run by us; no unaffiliated reproduction.
  • The soft-prompt result is one measurement on one layer of one checkpoint (Falcon-E). The source says the recipe was salvaged in weaker form and requires a step-0 logit-parity gate.
  • The T4 refutation rests on the documented requirement in a cited source, not on our own run. The source says the path survives only behind an UNVERIFIED torch<=2.6 / triton-3.2 / pinned-TRL environment.
  • The source notes there is no independent replication of the soft-prompt path anywhere.

06Outcome

Negative

The claim or approach failed, or we retracted it.

07What we learned

Several widely repeated recipe claims failed when we checked them, so we keep a public list of what failed. For each claim, the list says whether we ran it ourselves or checked it against a source's stated requirements.

08Check it yourself

Read the notes (link above).

  • L25
    Mixed

    RANGER mock run: low-bit quantization ideas tested on synthetic tensors

    On synthetic data, one sub-hypothesis was falsified and two design constraints emerged: rotate only after splitting off the sparse outliers, and rotation helps 4-bit activations but not below 4 bits. A later test showed width and rotation overlap on outliers. These are mechanism-level findings, not claims about real models.

    Q9

  • L26Undated in the source
    Mixed

    RANGER Phase 1: the quantization ideas on real models (SmolLM2-135M, Qwen3-0.6B)

    A prior null result was retracted publicly. Rotation recovery replicated across two architectures (92.3% and 92.5%), while the census-replaces-rotation shortcut was refuted, and a super-weight lever is null at 135M and only small at 0.6B. One of our own sub-results (0.38 rescue) was refuted by our confirmation pass.

    Q9

  • L32
    Null

    E-S1: how far can a ternary 2B model be compressed without loss?

    Pretrained ternary weights are close to uniform, so lossless coding gains little. Real savings would need changes to the model, not a better coder.

    Q9