b70-optimization-lab

Qwen3.6 27B INT4 AutoRound on B70

This folder is the result packet for the Intel/Qwen3.6-27B-int4-AutoRound lane on Intel Arc Pro B70.

Model Identity

The model card reports ten safetensor shards plus model_extra_tensors.safetensors; Hugging Face metadata reports about 17.71 GiB of tracked files.

Current Status

Initial TP1 single-B70 vLLM/XPU bring-up passed on 2026-07-03. The lane now has a strict fresh-response BF16-LM-head baseline, one validated env-only speed win, and faster quality-gated runtime INT8/draft-INT4 variants. LocalMaxxing approved the BF16-LM-head result as cmr4gokx90061nv01lhoe3ft8 and the runtime INT8-LM-head variants as cmr4zkcxb003yq9018408i1pn, cmr576apv0079q901i6dvsh0l, cmr5iu3gk00bfq901nidgcana, cmr8rg5d900glqr01g4fesy6i, and cmr9atqb800msqr01u760xh0t.

Current TP2 record:

The mechanisms matter: installed oneCCL Gold-2021.17.2 failed the exact BF16 [4,5120] XPUGraph all-reduce oracle on nearly every replay, while the pinned public revision passed direct 256/256 and graph 512/512 on both ranks. The draft then required a compiled all-gather custom-op boundary because Inductor’s functional wait_tensor cannot run inside an XPU command graph; direct BF16 [4,2560] all-gather capture passed 512/512 on both independent GPU pairs. Do not reproduce TP2 records against the known-broken installed collective or omit the draft-graph patch/env gate.

Validated so far:

Current Intel-checkpoint baseline valid fresh-response result:

Prior TP1 fastest quality-gated variant:

Previous fastest quality-gated variant:

Current prompt-processing / long-context service baseline:

Prior Intel-checkpoint quality-gated runtime-quantized variant:

Service-oriented scoped variant:

Post-GGUF recheck:

Current best synthetic diagnostic:

Current realistic research interpretation:

Next milestone: beat the runtime INT8-LM-head row with a real source change. The bounded retests after the record closed as no-promo: MTP depth remains best at k=3, including the later shallow-depth coverage pass where MTP1/cg8 51.246 and MTP2/cg8 59.589 both lost to MTP3/cg8 control 64.730 (../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-04-webhie-mtp1-mtp2-depth-coverage-no-win.md), current webhie/BF16-scale capture size remains best at cg8 (65.153 same-window control versus cg4 64.507, cg16 63.500, cg32 64.071; see ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-04-webhie-bf16scale-capture-size-screen-no-win.md), older target-only attribution was enough to justify the LM-head kernel lane, but the latest synchronized timing shows the active INT8 LM-head/local-argmax path is now small. The standalone native compact full-vocab top-1 kernel is exact but slower than dense oneDNN, and the semantic candidate-max version is also closed no-win: exact top IDs/values plus candidate scores, but only 1.010x at rows 1 and slower at rows 2-4. A low-level INT8 GEMM scratchpad ring-size screen is also closed no-promo: ring4 produced high support rows (65.708, 65.817) but crossover deltas versus ring1 controls were only +0.42% and +0.27%; see ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-04-int8-gemm-scratchpad-ring-screen-no-win.md. A later producer-side INT4 draft-LM-head top-1 prototype also closed no-win: the sycl8 build passed top-id correctness, but full-vocab rows 1..4 were slower than dense logits plus argmax (2.30/5.82/6.52/9.15 ms vs 1.95/1.37/1.21/1.22 ms); see ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-07-int4-top1-prototype-sycl8-no-win.md. The next meaningful decode-rate work is improving accepted/generated tokens per target verifier step, finding a stronger fresh-request draft source, reducing target-forward cost in the Qwen3.5/Next body, or building a graph-safe exact GDN/spec-state transaction that lets stronger drafting remain correct. LM-head producer work is now a secondary cleanup unless it is a genuinely new backend primitive with microbench evidence. The follow-up source audit in ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-04-lmhead-callcount-source-audit.md narrows that further: small Python-level row/call shortcuts are unlikely to win because they still use dense LM-head primitives. Future Qwen27 work should start with a real fused/top-ID LM-head primitive, a native row-adaptive verifier, or a materially stronger target-matched drafter, not another config sweep. The latest frontier audit also tested oneDNN Graph MatMul -> ReduceMax directly: BF16 stayed as two one-op partitions and the tested INT8 graph form was rejected, so there is no cheap oneDNN Graph wrapper shortcut to promote. See ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-04-frontier-audit-onednn-graph-and-drafter.md. The 2026-07-06 branch/regenerate trace probe added default-off VLLM_XPU_BRANCH_REGEN_TRACE=1, repaired the local XPU runtime after sycl9 _C/_moe_C/FA2 binaries broke device inference and FA2 varlen_fwd, and completed a strict fresh diagnostic row at 65.078 tok/s with cached_tokens=0 on every prompt. The trace summarized 220 scheduled verifier rows: 1.6727 mean accepted draft-prefix tokens, 2.6727 mean raw visible tokens, 39.09% full accept, and 292 branchable remaining draft rows after partial rejects. Treat this as branch/tape infrastructure evidence, not a headline result: by itself the measured MTP3 branch surface is too narrow to be the primary 125+ tok/s path. See ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-06-branch-regen-trace-probe-and-sycl8-restore.md. The 2026-07-06 draft-side mtp.fc runtime INT8 experiment is also closed: it targeted only the BF16 Qwen3.5 MTP mtp.fc layer and preserved exact target verification, but the completed candidate (66.777 tok/s) lost to same-window controls (67.954 and 67.994 tok/s), while another candidate exposed a TorchDynamo fake-tensor unsupported-op failure for _xpu_C.int8_gemm_w8a8 inside compiled MTP. See ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-06-mtp-fc-int8-no-win.md and ../../patches/qwen36-27b-autoround-int4-b70/vllm-qwen27-mtp-fc-int8-no-win-20260706.patch. The GDN gated-RMSNorm rstd skip is also closed no-win: skipping an ignored Triton rstd allocation/writeback via VLLM_XPU_RMSNORM_SKIP_RSTD=1 produced 66.329 and 66.595 tok/s, below same-window controls at 67.716 and 67.910 tok/s; see ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-06-rmsnorm-skip-rstd-no-win.md and ../../patches/qwen36-27b-autoround-int4-b70/vllm-qwen27-rmsnorm-skip-rstd-no-win-20260706.patch. The DFlash mixed-SWA audit is captured in ../../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-04-dflash-mixed-swa-multikv-blocker.md: the target runner is multi-KV aware, but the DFlash/EAGLE drafter path assumes one KV group, so deleting the assertion would risk invalid draft-cache writes. The later mixed-SWA implementation attempt and Hipfire feasibility closure found the real mixed DFlash draft was not accepting enough tokens on this fixed suite, so do not reopen it for record chasing without a stronger draft. Keep synthetic screens for candidate search only, then rerun the Qwen realistic suite with --return-token-ids and the quality suite before promotion.

First diagnostic realistic-suite run (not a headline result):

Start Here

Initial Hypotheses

  1. TP1 should fit on a single 32 GB B70 at short context because the checkpoint is INT4 AutoRound with FP16 exceptions, not full BF16.
  2. The fastest research loop should use four independent TP1 replicas on the four B70s, mirroring the Gemma workflow, before considering TP2/TP4.
  3. Built-in qwen3_next_mtp may provide an early speedup, but headline claims must still use fresh prompts with cached_tokens=0 and target-verified accepted tokens.
  4. Loader support is the first risk: local vLLM must correctly map quant_method=auto-round / auto_round:auto_gptq to an XPU-supported W4A16 path without silently dequantizing or CPU fallback.

Claiming Rules

This site is open source. Improve this page.