Aefinity AI

L17 Mixed Q6 · Do our tests test anything

E27: exp, ln and SiLU on every float32 input, and a defect in our spec

QuestionIs the pinned exponential accurate on every possible input, and are the spec's safety guards correct?

Run by us; no unaffiliated reproduction

01Why it matters

The softmax inside every transformer uses exp. If its guards are wrong the spec pins a wrong answer exactly.

02Method, step by step

  1. Evaluate exp, ln and SiLU on all 2^32 float32 bit patterns.
  2. Compare against a float64 reference and run oracle-free monotonicity checks.
  3. Re-run on a second machine.

Method as recorded

Every one of the 2^32 f32 bit patterns evaluated for exp_pinned, ln_pinned and silu_pinned under the pinned environment (FTZ/DAZ, 8 worker threads) against a host glibc float64 oracle narrowed to f32 and flushed, plus oracle-free monotonicity checks. Run by us on one machine (an x86_64 VM); the same binary was re-run by us on a second machine (Celeron, no AVX2, same glibc) with a byte-identical log.

03Results, exact and sourced

  1. R1Every one of the 3,257,925,634 arguments where both the pinned exp and the oracle are finite is within one ULP (max relative error 1.1921e-7 = 2^-23), measured against a float64 oracle.

    Every one of the 3,257,925,634 arguments for which both `exp_pinned` and the oracle are finite is within **one ULP**.

    Verbatim source cis2-spec / E27_EXP_LN_RANGE.md

  2. R2Defect in our own spec: the high guard (x > 88.0 returns +Infinity) clips 94,743 arguments (0x42B00001 to 0x42B17217) whose true exp is finite and representable in f32. Behaviour is pinned as normative and unchanged; an erratum was filed.

    §6.2's high guard clips 94,743 values below the representable range

    Verbatim source cis2-spec / E27_EXP_LN_RANGE.md

  3. R3exp and ln were monotone over the whole finite domain (zero violations), an oracle-free check.

    exp monotone: violations=0

    Verbatim source cis2-spec / E27_EXP_LN_RANGE.md

04What it does NOT show

Correct rounding, a second compiler or libm, or that the defective range is reachable by other models or prompts (it was entered 0 times in the instrumented decode).

05Scope and caveats

  • Accuracy is against a float64 oracle (host glibc exp narrowed to f32), not correct rounding; where narrowing double-rounds the oracle itself may differ from the correctly rounded f32.
  • The headline is the guard defect, not the accuracy.
  • The source states the second-machine re-run used the same binary and same glibc, so it does not test a second compiler or libm.
  • The clip band was entered 0 times in an instrumented decode (one model, six ASCII prompts, at most 64 tokens); no claim that other models or prompts cannot reach it.
  • Two-ULP silu figure is for the function in isolation, not accumulated through an FFN.
  • The source says an earlier draft's FTZ reasoning about the silu discontinuity was wrong; E39 (CHANGELOG ERRATUM E-12) later corrected neighbouring FTZ reasoning.
  • Run by us; no unaffiliated reproduction.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

The accuracy was the unremarkable part. Exhaustive testing found a real defect in our own spec that sampled tests could not, and we recorded it in the open.

08Check it yourself

Read the page (link above).

  • L05
    Mixed

    Sin/cos accuracy for RoPE against a pre-registered 2 ULP bar (v0.2, v0.3, v0.3b)

    Fixing only the reduction improved the error a lot and still failed the bar, because the polynomial was now the dominant error. We kept the bar and changed both the reduction and the polynomials, which met it in the local harness.

    Q6

  • L13
    Mixed

    E22: which spec rules does the test vector actually exercise?

    Passing the published test does not prove every rule matters. 12 of 16 single-rule breaks were caught. The tie-break and the denormal-flush pin are protected only by inputs this vector never contains. One of our own null results turned out to be a vacuous edit that tested nothing, and we corrected it in place. Correcting it exposed a weak spot in the spec's own rsqrt check, which tests only a value where every implementation agrees.

    Q6

  • L15
    Positive

    E25: the RMSNorm multiply order matters, and our spec sentence saying otherwise was false

    A reordering that leaves the generated text unchanged (on this short vector) still changes the bits, so comparing text is not enough. A published limitation was out of date, and we corrected it in the open.

    Q6