Aefinity AI

L05 Mixed Q6 · Do our tests test anything

Sin/cos accuracy for RoPE against a pre-registered 2 ULP bar (v0.2, v0.3, v0.3b)

QuestionIs the pinned sin/cos accurate enough for long contexts, measured against a bar we set before we tested?

Run by us; no unaffiliated reproduction

01Why it matters

Rotary position encoding feeds sin and cos with large angles at long context. If the pinned functions are inaccurate there, the spec pins a bad answer exactly.

02Method, step by step

  1. Write the pass mark down first: within 2 ULP of the correctly rounded f32 value.
  2. Build a 60-digit-precision reference (mpmath) for the angles up to position 8192 on two models.
  3. Score three versions: the original, a version with a higher-precision reduction only, and a version with octant reduction plus new polynomials.
  4. Report each against the bar.

Method as recorded

A pre-registration set a 2 ULP bar (oracle within 2 ULP of correctly-rounded f32) before the v0.3 fix was implemented. An mpmath 60-digit oracle covered RoPE angles for positions up to max_position_embeddings=8192, using the SmolLM2-135M (theta=1e5) and Qwen2.5-0.5B (theta=1e6) values. It compared three versions of the sin/cos reduction. The v0.3b mechanism was added by a coordinator directive after v0.3 missed the bar. Local x86_64 sandbox harness (scripts/h2m_accuracy_harness.py). Run by us.

03Results, exact and sourced

Maximum sin/cos error in ULP against the pre-registered 2 ULP bar (log scale)
v0.2 sin115,198 ULP
v0.2 cos163,969 ULP
v0.3 (max)2,813.5 ULP
v0.3b sin1.5 ULP
v0.3b cos1.5 ULP
1101001k10k100k1M2 ULP bar

v0.2 sin: 115198 ULP; v0.2 cos: 163969 ULP; v0.3 (max): 2813.5 ULP; v0.3b sin: 1.5 ULP; v0.3b cos: 1.5 ULP; bar: 2 ULP

The values come from different local runs (v0.2 from the oracle run, v0.3b from the harness run, v0.3 as raw ULP), so the bars are not a like-for-like comparison. v0.3b meets the bar; v0.2 and v0.3 do not. Figure drawn from the sourced results below.

  1. R1The pre-registration was written before implementation.

    **Status: pre-registered before implementation.**

    Verbatim source cis2-spec / E15m_PREREG.md

  2. R2v0.2 (f32-only reduction, Taylor polynomial) had errors up to 115198 ULP for sin and up to 163969 ULP for cos, in the local oracle run.

    | v0.2 (f32-only reduce, Taylor) | up to 115198 | 1.30e-2 | up to 163969 | 1.36e-2 |

    Verbatim source cis2-spec / E15m_RESULT.md

  3. R3v0.3 (f64-staged reduction only) cut the error by roughly 55 to 70 times, with a max of 2813.5 raw ULP, and still missed the 2 ULP bar. The doc attributes the residual to the unchanged Taylor polynomial.

    "<=2 ULP" bar is NOT met

    Verbatim source cis2-spec / E15m_RESULT.md

  4. R4v0.3b (octant reduction plus separate minimax polynomials) reached a maximum of 1.5 ULP for sin and cos on both models and met the bar, in the local harness run.

    | **v0.3b (octant reduce, minimax)** | **1.5** | **1.2e-7** | **1.5** | **1.2e-7** |

    Verbatim source cis2-spec / E15m_RESULT.md

04What it does NOT show

Cross-ISA or CI confirmation (pending when written), positions above 8192, or that the final mechanism was pre-registered. It was added after the second version missed.

05Scope and caveats

  • Local x86_64 sandbox harness numbers only. CI runs were dispatched but were still in flight when the document was written, and the doc records no CI confirmation. Cross-ISA (aarch64) confirmation was pending.
  • Positions 0 to 8192 only. The v0.3b run used a dense grid (stride 7, plus positions 0 to 19 and the last position), not every position.
  • The 2 ULP bar was pre-registered for the v0.3 mechanism. The v0.3b mechanism was chosen after v0.3 missed, so it was not itself pre-registered.
  • The doc says the v0.3 ULP counts are inflated near sin/cos zero-crossings, a known property of the ULP metric. The v0.3 filtered figure (values with |sin/cos| >= 1e-2) is up to 486.5 ULP.
  • The work sits on branch cm/e15m-spec-v0.3-rope with 'Do NOT merge to main pending review'. The doc is on main, but the branch URL returns 404 unauthenticated, so the branch work is not public.
  • Gate (3), the token-match oracle runs, was not re-run for v0.3b in that pass.
  • Run by us; no unaffiliated reproduction.

06Outcome

Mixed

Some parts held, some did not, or we corrected ourselves.

07What we learned

Fixing only the reduction improved the error a lot and still failed the bar, because the polynomial was now the dominant error. We kept the bar and changed both the reduction and the polynomials, which met it in the local harness.

08Check it yourself

Read the pre-registration and the result page (links above). The branch the work sits on is not public.

  • L13
    Mixed

    E22: which spec rules does the test vector actually exercise?

    Passing the published test does not prove every rule matters. 12 of 16 single-rule breaks were caught. The tie-break and the denormal-flush pin are protected only by inputs this vector never contains. One of our own null results turned out to be a vacuous edit that tested nothing, and we corrected it in place. Correcting it exposed a weak spot in the spec's own rsqrt check, which tests only a value where every implementation agrees.

    Q6

  • L15
    Positive

    E25: the RMSNorm multiply order matters, and our spec sentence saying otherwise was false

    A reordering that leaves the generated text unchanged (on this short vector) still changes the bits, so comparing text is not enough. A published limitation was out of date, and we corrected it in the open.

    Q6

  • L16
    Mixed

    E26: sin/cos at very long contexts — the sampled answer was wrong

    4,096 samples per binade never hit the worst points among about 8.4 million, so sampling gave a comfortable answer that was wrong. An exhaustive sweep replaced it. Absolute error stays flat while ULP error grows near zeros of sin/cos, so ULP alone is the wrong figure of merit for these functions.

    Q6