Aefinity AI

L22 Mixed Q10 · Repeatable evaluation

ev-1: a 160-item mini evaluation on the pinned 135M model

QuestionCan a tiny model be evaluated in a way anyone can repeat exactly, and what does it actually score?

Run by us; no unaffiliated reproduction

01Why it matters

Before measuring big claims we wanted a small evaluation anyone can repeat bit for bit.

02Method, step by step

  1. Assemble 160 items in four categories.
  2. Run them through the repository's reference binary on SmolLM2-135M, fp32, greedy.
  3. Score with simple substring and refusal-marker heuristics.
  4. Run twice and compare digests.

Method as recorded

160 items in four categories (60 truthfulness sampled from TruthfulQA, 20 self-consistency, 60 tool-use arithmetic-completion proxy, 20 refusal probes) on SmolLM2-135M base (fp32, greedy) through the repo's own cis2_ref reference binary. Heuristic substring and refusal-marker scoring. Run by us on a single host. This is a substitute for the originally intended 2B model, which is not in this repo.

03Results, exact and sourced

Items passed, by category
Truthfulness3 / 60
Tool-use proxy5 / 60
Refusal probes10 / 20
Self-consistency20 / 20

Truthfulness: 3 of 60; Tool-use proxy: 5 of 60; Refusal probes: 10 of 20; Self-consistency: 20 of 20

Self-consistency measures repeatability, not accuracy. The source says the refusal score sits at chance. Figure drawn from the sourced results below.

  1. R1Truthfulness: 3 of 60 passed (0.05), by verbatim-substring heuristic.

    | `truthfulness` | 60 | 3 | 0.05 |

    Verbatim source cis2-spec / 2026-09-22-ev1-mini-eval-kit.md

  2. R2Refusal probes: 10 of 20 passed (0.5). The source says this sits at chance and was not broken out further.

    | `refusal` | 20 | 10 | 0.5 |

    Verbatim source cis2-spec / 2026-09-22-ev1-mini-eval-kit.md

  3. R3Tool-use proxy (arithmetic completion, not tool-call format): 5 of 60 passed.

    | `tool_use` | 60 | 5 | 0.0833... |

    Verbatim source cis2-spec / 2026-09-22-ev1-mini-eval-kit.md

  4. R4Self-consistency: 20 of 20 items produced identical run1/run2 digests within the same process on one host. A separate cross-process rerun log exists. No second-host verify was run.

    | `self_consistency` | 20 | 20 | 1.0 |

    Verbatim source cis2-spec / 2026-09-22-ev1-mini-eval-kit.md

  5. R5The committed summary.json matches the four category numbers above.

    "truthfulness": { "n_items": 60, "n_pass": 3,

    Verbatim source cis2-spec / summary.json

04What it does NOT show

Anything about a 2B or instruction-tuned model, safety, or other machines. The second-host verify was not run.

05Scope and caveats

  • A non-instruction-tuned 135M base model; truthfulness and tool-use scores are near the floor with weak heuristic scoring.
  • Model substitution: this is NOT the 2B model; no claim is made about any 2B-scale or instruction-tuned model.
  • Refusal items are narrow, non-operational probes written by us; refusal 10/20 is at chance and is not a safety measurement.
  • tool_use is arithmetic completion, not tool-call-format correctness.
  • Single host. The standalone --mode verify pass on a second machine has NOT been run, so there is no cross-host claim.
  • The point is repeatability, not accuracy.
  • Run by us; no unaffiliated reproduction.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

The model scores poorly but repeats exactly on one machine. We report the poor accuracy alongside the perfect same-host repeatability. Cross-host verification is still open.

08Check it yourself

Read the page and the committed summary.json (links above).