b70-optimization-lab

DeepSeek V4 K160 MTP1 M=2 W8A16 record

Date: 2026-07-15

Outcome

Extending the already-qualified selective W8A16 projection path from M=1 to the M=2 target verifier produces a new record:

• strict headline: 54.464909 tok/s median, 51.586812 p10;
• strict confirmation: 54.445287 tok/s median, 51.684740 p10;
• prior row-exact MTP1: 50.016860 tok/s;
• corrected nonspeculative base: 40.170350 tok/s;
• gain: 8.89% over prior MTP1 and 35.59% over base;
• measured acceptance: 1,556/2,003, or 77.68%;
• sustained exact gate: 20/20, including ten after both strict suites;
• every request reports cached_tokens=0.

LocalMaxxing approved cmrmfivhg1nmamj012e3138my.

Evidence:

/mnt/fast-ai/bench-results/deepseek-v4-flash-xpu/mtp1-rowexact-w8a16-m2-candidate-20260715T1945Z

Mechanism

The selective W8A16 record lane already uses BF16 activations with FP8 weights for four dense projection families at M=1:

• fused WQA/WKV: N1536, K4096;
• Q-B: N8192, K1024;
• O-B: N4096, K2048;
• shared gate/up: N1024, K4096.

The launcher previously fixed VLLM_XPU_V4_BLOCK_FP8_W8A16_MAX_M=1, so MTP1 target verification at M=2 fell back to the W8A8 path. Setting the limit to 2 lets oneDNN consume both verifier rows in one W8A16 operation.

On every B70, all four production shapes matched two independent M=1 W8A16 calls bit-for-bit over 40 changing epochs per shape. One M=2 call was 2.42-2.50x faster than the pair. Raw card artifacts are:

/mnt/fast-ai/bench-results/deepseek-v4-flash-xpu/w8a16-m2-row-invariance-card{0,1,2,3}-20260715.json

This is a real verifier optimization, not an acceptance change: acceptance remains approximately 77.7%, matching the prior MTP1 lane. It reduces the target’s dense projection cycle while preserving row arithmetic.

Quality and runtime identity

The M=2 compressor remains row-exact; this candidate does not re-enable the batched compressor path that previously leaked prompt text. The exact gate is split around the strict load: ten captures before and ten after. All four workers map the selected exact-version oneCCL runtime and matching SYCL/UR libraries. The large-prefill all-reduce route remains 128 KiB.

• vLLM: 93fde4186b34d61e2d31ee258be282a8f768f3ed;
• XPU kernels: de979b9328408302129c10ba04ed2fe090d414df;
• oneCCL: 6da44bcb540f1005de88276dd86465cf5de4dc2f;
• libccl SHA-256: bf9f2e447ebb8373d7b0a41b15a32f2518a670a107d6f8fd84474798d8f2301b.

Decision

Promote and keep the candidate live on 127.0.0.1:18080. Do not widen W8A16 to unmeasured shapes or M>2. MTP2 reuse remains closed. The next credible candidate should target remaining M=2 W8A8/MXFP4 verifier work or sampler coordination, with an exact row-invariance microgate before a server load.

This site is open source. Improve this page.