b70-optimization-lab

Laguna exact M=8 BF16-input router top-k preregistration

Date registered: 2026-07-23 America/Toronto

Status at registration: the default-off source candidate was under static review. No candidate binary had been built, no component GPU gate had run, and no endpoint service had been launched.

Question and arithmetic contract

The retained profile attributes 0.272194 ms per 47-layer target cycle to materializing BF16 router logits as FP32 and 0.560374 ms to TopKGating. The candidate removes only the materialized BF16-to-FP32 tensor. It must preserve the incumbent operation:

BF16 [8,256] logits
  -> exact widening to FP32
  -> FP32 sigmoid
  -> add FP32 [256] correction bias for selection only
  -> lower-expert-ID tie break
  -> top 10
  -> renormalize the unbiased FP32 sigmoid weights

The router-local scaling factor is exactly 1.0; Laguna’s configured 2.5 scale is applied later by the MoE runner because apply_routed_scale_to_output=True. Router softcapping is zero. The 47 checkpoint correction-bias tensors are FP32 [256] tensors and are all exactly zero, but the component gate also uses changing nonzero FP32 biases.

The candidate is a separate _moe_C entry point. It does not change the generic BF16 top-k behavior, whose sigmoid is rounded through BF16 and is not an exact substitute for the incumbent FP32 path.

To preserve the incumbent expert-to-lane mapping, the specialization is fixed to four warps per workgroup and an eight-byte BF16 load. Each of 32 lanes owns the same eight experts as the FP32 kernel:

lane l: 4*l .. 4*l+3 and 128+4*l .. 131+4*l

The XOR reduction order remains 16, 8, 4, 2, 1, with two four-warp workgroups for eight rows.

Selector and fail-closed scope

The default-off selector is:

VLLM_XPU_LAGUNA_M8_BF16_ROUTER_TOPK=1

It may skip router_logits.float() only for Laguna target verifier execution at exactly M=8 under the existing exact batched-M1 path. It must require 256 experts, top 10, sigmoid routing, normalization, an FP32 correction bias, router-local scale 1.0, no softcap, no EPLB/redundant/hash routing, and the specialized extension symbol.

A matching flagged M=8 call must raise on contract or binary drift. M=1, M=2..7 verifier tails, prefill, DFlash draft layers, non-Laguna models, and all other shapes keep the incumbent FP32-materialization path.

Source frozen before build:

Focused build and binary freeze, completed before the first component GPU process:

Both binaries and the symbol/compiler checks are preserved under the build-log directory above.

Static gate freeze:

Four-card component gate

Run the tracked component harness once on each physical B70 with exactly one visible device. Every card receives the same 333 changing epochs:

The production fixtures must be formed with the incumbent batched-M1 BF16 gate projection from the retained [8,3072] mlp-input traces and the exact checkpoint gate weight for that layer. Their manifest and hashes are frozen as gate evidence. A missing or differently hashed production fixture is a failure, not permission to run a synthetic-only gate.

For each epoch, compare:

Require raw-byte equality and torch.equal for FP32 weights, int32 expert IDs, and int32 token/expert source indices. Also require candidate repeat determinism, unchanged logits and bias, ten distinct in-range IDs per row, the exact slot * 8 + row source mapping, and the intended lower-ID winners in designed ties. The specialized dispatch must be proven rather than inferred.

After correctness, time only cast plus top-k for A and direct top-k for B. Exclude gate projection, fixture generation, hashing, and CPU work. Reuse output buffers and synchronize only at arm boundaries:

Every physical card must independently satisfy all of:

One mismatch, nondeterministic repeat, failed card, or missed timing threshold stops this lane before an endpoint. Results are not averaged across cards to mask a failure.

Component result

The frozen component gate passed on all four physical B70s. Each card passed all 333 pre-timing changing epochs, 1,998 torch.equal checks, 2,664 raw-byte checks, candidate-repeat determinism, unchanged-input checks, explicit lower-expert-ID ties, and the 111-epoch post-timing replay. Synthetic, production-source, projected production-fixture, and aggregate output hashes were identical across cards.

All cards won 31/31 paired A-B-B-A blocks:

Rank A cast+top-k ms/cycle B direct BF16 ms/cycle Saved ms Gain
0 0.938074 0.456029 0.482089 51.3965%
1 0.911805 0.455101 0.456650 50.0925%
2 0.922209 0.454956 0.467250 50.6665%
3 0.906338 0.454877 0.451434 49.8086%

This clears every preregistered component threshold and authorizes the cold endpoint A-B-B-A protocol. Raw evidence:

/media/steve/CorsairExternal/llm-optimization-artifacts/laguna-s-2.1/logs/m8-bf16-router-topk-component-689ee36-af68118-20260723T043723Z/

Compact tracked result:

data/laguna-s-2.1-m8-bf16-router-topk-component-20260723.json

Endpoint protocol and early stop

Only if all four component gates pass, use the approved eager depth-7 record stack with route-parallel W2 and expert N-tile interleave enabled and QKNorm, attention-MM, graph, and unrelated candidates disabled. Change only the new router selector.

Use the fixed 13-prompt suite with SHA256 9fdaacfdc4de59407a73cbe0d8130fa0f6abe91fed782e399a58adbc035ea638, the canonical q=1 teacher, seed 1, BF16 KV, enable_thinking=false, max_tokens=512, token IDs returned, one active request, and a new cold service for every leg. There is no generation warm-up, repeated prompt, prefix/history/ngram reuse, concurrent request, or fifth rescue run. Keep all four devices free for 60 seconds between legs.

The order is sequential A-B-B-A:

  1. A1, selector off;
  2. B1, selector on;
  3. B2, selector on only if the phase-1 gates pass; and
  4. A2, selector off.

Stop after B1 unless:

Endpoint tooling was frozen before A1:

The runner rejects ambient benchmark-sensitive vLLM, Laguna, Level Zero, SYCL, Unified Runtime, oneCCL/libfabric, CPU-threading, compiler, and loader variables. It then pins the approved route-interleaved stack, model/draft revisions, tokenizer, chat template, model index/code, runtime packages, driver tooling, source commits, and native binaries. Every leg byte-hashes all LFS weights against the frozen Hugging Face revision manifests before starting the service. An independent pre-run audit matched all 16 LFS files: 15 target shards plus the DFlash weight, 74,137,878,672 bytes total, with zero mismatches in 39.863 s.

The runner also refuses path reuse/traversal, validates exactly four idle B70 devices before startup, and performs bounded process-group shutdown plus port/process/four-device idle proof on both successful and failed legs.

The first A1 invocation with runner c1a58a0bec4869190183a32061b1f2c24f96e79a6ee69900bf2310a19b6087d2 stopped inside the XPU idle preflight because the local mawk rejected a multiline parenthesized assignment. It exited before the service, model, or any endpoint generation started. The failed preflight is preserved at:

/media/steve/CorsairExternal/llm-optimization-artifacts/laguna-s-2.1/runs/m8-bf16-router-topk-abba-689ee36-af68118-20260723T051304Z/01-A1-control/

Only that header-expression syntax was made mawk-portable. The frozen preflight capture then passed the exact validator, the analyzer’s runner hash was updated, and a new run directory is required for the real A1.

Quality, attribution, and record gates

Every executed leg must have 13/13 full token arrays bitwise equal to the canonical q=1 teacher, 13/13 cached_tokens=0, one request per unique cold prompt, 512-token long-then-next exact 2/2, rollover exact 1/1, and exact cross-leg token equality. Record draft cycles, drafted/accepted totals, accepted-position histograms, per-row timings, and target-cycle-normalized decode time.

If the full block runs, call it a reproducible endpoint win only if B1 beats A1 and B2 beats A2 in headline throughput, each candidate wins at least 9/13 rows with positive median paired change, each saves at least 0.15 ms per target cycle, each pair’s acceptance rate differs by no more than 0.10 percentage point, and the lower candidate exceeds the lower control.

A LocalMaxxing record additionally requires the lower candidate to exceed the approved 33.438926675602126 tok/s record. Only that lower candidate may be submitted after a complete payload and evidence audit. Otherwise preserve the candidate as an exact win, loss, or inconclusive result with no submission.

Endpoint phase-1 result

The real A1/B1 phase ran under the frozen runner and analyzer at:

/media/steve/CorsairExternal/llm-optimization-artifacts/laguna-s-2.1/runs/m8-bf16-router-topk-abba-689ee36-af68118-20260723T051657Z/

Both legs passed every quality, honesty, benchmark-identity, stack-identity, freshness, and cleanup check. Each was 13/13 bitwise exact against the canonical q=1 teacher with all 13 cached-token counts zero. The 512-token long-then-next check passed 2/2 and the rollover check passed 1/1 in each leg. Because both legs exactly matched the same teacher, their output token arrays also matched one another. There was no generation warm-up, prompt reuse, history acceleration, concurrent request, or retained endpoint.

The measured phase-1 comparison was:

Metric A1 control B1 candidate B1 - A1
Headline median tok/s 32.9690119264 32.3101222212 -0.6588897052 (-1.9985%)
Mean tok/s 38.3387134160 38.6575563099 +0.3188428939
p10 tok/s 25.6497789812 26.4776872145 +0.8279082334
Aggregate decode seconds 159.2497690071 157.7562722090 -1.4934967981
Target-cycle-normalized time 92.7488462476 ms 91.7187629122 ms -1.0300833354 ms (-1.1106%)
DFlash cycles 1,717 1,720 +3
Draft tokens 12,019 12,040 +21
Accepted tokens 4,645 4,642 -3
Acceptance rate 38.6471420251% 38.5548172757% -0.0923247494 percentage point

The candidate won 10/13 paired prompt rows. Its median paired improvement was +0.2127728662 tok/s (+0.7138466435%), and it passed the row-win, paired-median, cycle-saving, and acceptance-delta gates. The three losses were the median-defining shell-safety-review row (-1.9985%), structured-extraction (-2.3034%), and prose-decision-memo (-1.7132%). Consequently the preregistered B1 headline > A1 headline gate failed.

The frozen analyzer classified the experiment phase1_failed_stop. Per the preregistered early-stop rule, B2 and A2 were not run. This is evidence that the specialization removes real work, but it is not a promotable endpoint win under the frozen official metric. No LocalMaxxing payload was staged or submitted.

Tracked summary:

data/laguna-s-2.1-m8-bf16-router-topk-endpoint-phase1-20260723.json

Artifact hashes:

This site is open source. Improve this page.