Aefinity AI

L03 Positive Q2 · Graphics cards and shortcuts

Batching and precision on a Tesla T4: which shortcuts change the digest

QuestionDo common GPU speed shortcuts, like batching requests or switching to int8, quietly change the answer?

Run by us; no unaffiliated reproduction

01Why it matters

Batching requests and lower precision are the usual ways to make serving cheaper. If they silently change answers, a receipt must be able to tell.

02Method, step by step

  1. Take one matrix multiply from the reference model (layer 0, q_proj).
  2. Compute it on a Tesla T4 and on a CPU build in three ways: row-batched fp32, grouped fp32 and int8, at batch sizes 1, 4 and 8.
  3. Count how many of the 576 output values differ from the batch-1 CPU result.
  4. Compare host and GPU digests for all 10 configurations.

Method as recorded

One matmul (q_proj, layer 0, hidden size 576) of the reference model on a Kaggle Tesla T4 (sm_75) and a host CPU build (g++; CUDA build via nvcc for sm_75): row-batched fp32, grouped fp32 and int8 at batch sizes 1, 4 and 8, plus a batch-1 host control (10 configurations). Diffs are counted over 576 output elements against the batch-1 CPU reference. Run by us; driver source and recipe not published. No timing figures are reported.

03Results, exact and sourced

Elements that differ from the batch-1 reference, out of 576
Row-batched fp32 (batch 1, 4, 8)0 / 576
Grouped fp32455 / 576
int8576 / 576

Row-batched fp32 (batch 1, 4, 8): 0 / 576; Grouped fp32: 455 / 576; int8: 576 / 576

One matrix multiply on a Tesla T4; host and CUDA builds agreed on all 10 configurations. Figure drawn from the sourced results below.

  1. R1Row-batched fp32 matched the batch-1 reference exactly at every batch size, on host and GPU.

    fp32-seq matched the batch=1 reference exactly at batch sizes 1, 4, and 8 (0/576 diffs each), on both the host build and the T4 CUDA build.

    Verbatim source cis2-spec / 2026-09-19-GPU-BATCHED-FP32-DETERMINISM.md

  2. R2Grouped fp32 and int8 changed the result at every batch size tested, as the report predicted before the run (455/576 and 576/576 differing elements respectively).

    fp32-grouped and int8 mismatched at every batch size tested (455/576 and 576/576 diffs respectively)

    Verbatim source cis2-spec / 2026-09-19-GPU-BATCHED-FP32-DETERMINISM.md

  3. R3Host and CUDA builds agreed on all 10 configurations.

    Host and CUDA builds produced bit-identical digests for every one of the 10

    Verbatim source cis2-spec / 2026-09-19-GPU-BATCHED-FP32-DETERMINISM.md

04What it does NOT show

Behaviour of a whole model, other GPUs, or whether grouped or int8 could be made to match.

05Scope and caveats

  • Single matmul (q_proj, layer 0), not a full forward pass; the whole-model follow-on covers only row-batched fp32 and is in an internal log, not a public report.
  • One GPU architecture (Tesla T4, sm_75); no claim about other GPUs or compute capabilities.
  • Whether grouped or int8 could be made to match was not attempted; those mismatches were the expected outcome.
  • Driver source and Kaggle recipe not published, so the sweep cannot be re-run from public material.
  • The published sweep table lists host-build digests; the CUDA digests are reported as byte-identical to them, not tabulated separately.
  • Run by us; no unaffiliated reproduction.

06Outcome

Positive

The check worked as expected, within the stated scope.

07What we learned

In our fp32 test, the pinned order of operations decided the digest. The same order matched on CPU and GPU, and a different order broke it, which is what a digest is supposed to detect. This is one matmul on one GPU type, so it shows the check works there, not that it holds everywhere.

08Check it yourself

Read the result page. The driver source and Kaggle recipe are not published, so the sweep cannot be re-run from public material.