b70-optimization-lab

B70 Optimization Lab Docs

This docs folder is the navigation layer for a multi-model Intel XPU optimization lab. Executable recipes live under ../repro/, promoted or closed-out model packets live under ../results/, active research lanes live under ../experiments/, and chronological evidence lives under ../notes/. Docs should point to those artifacts instead of duplicating every script.

Start Here

Model Lane Entry Points

Community And Operations

Build Photos

The community build guide includes example B70 photos and explains what details future contributors should capture: card spacing, airflow, power, slot order, risers, cooling, and visible diagnostics.

Repository Layout

Hardware Coverage

This lab currently has four Arc Pro B70 32 GB GPUs, so the total local Intel VRAM budget is 128 GB and every card is often occupied by active optimization runs. That is a useful community baseline, but it also means larger model families, larger context windows, and concurrent A/B lanes wait behind the same four devices. Additional high-VRAM Intel hardware, including Crescent Island-class 160-480 GB evaluation devices if available, would turn many of the open vLLM/XPU and driver/runtime questions here into directly measurable lanes rather than capacity-constrained TODOs. There is also spare EPYC 9015 platform capacity with up to ten PCIe 5.0 x16 slots, so larger Intel GPU coverage would have an immediate place to land.

How To Help

The best help is evidence that is easy to reuse: exact commands, driver/runtime versions, model identity, quality checks, logs, and negative results. This project has already turned B70 runs into LocalMaxxing submissions, X discussion, GitHub-indexed troubleshooting, and reusable vLLM/XPU and llama.cpp notes. To increase the number of optimized models, the highest-leverage additions are more independent Intel test systems, larger-VRAM Intel devices, clean driver/runtime repros, and help turning local findings into upstream issues or patches.

Deployable Baselines And Current Frontiers

There are multiple useful “current” paths depending on the task. Treat these as entry points, not a single global winner.

MiniMax 32K Deployable Endpoint

The clean “start from Ubuntu 24 and serve on the LAN” baseline is:

The served endpoint was also validated at prompt 32,408 / output 64 without OOM, and warm short decode stayed near 84.1 output tok/s afterward. The older strict speed record and newer constrained structured-output lane remain useful references, but they were measured on different conditions. The current host’s PCIe4 fabric measured about half the old large-message allreduce bandwidth, which is a plausible reason this fresh deployment lands at 83 output tok/s instead of the older 89-93 class. See ../notes/2026-05-23-current-host-pcie4-prefill-check.md for the math.

The 32k context promotion is documented in ../notes/2026-05-23-b70-display-disable-32768-context.md.

The session-cache, TurboQuant, and full-context research work is indexed in current-reproducibility-map.md and ../experiments/minimax_xpu_kv_offload/REPRODUCE.md.

Qwen3.6 35B Reference Packet

The Qwen3.6 35B lane is indexed in qwen36-research-map.md, with its final result packet in ../results/qwen36-35b-quark-int8-b70/.

Qwen3.6 27B INT4 AutoRound

The active Qwen27 one- and two-B70 optimization lane is indexed in ../results/qwen36-27b-autoround-int4-b70/. The current strict fresh-response record is 95.384868 tok/s on TP2. It uses graph-safe Intel FlashAttention to capture one full four-row target graph plus exact ReplaySSM pending/direct-output transaction fusions, and passed exact cases, repeat128, baseline parity, and the 1K needle. It uses the webhie AutoRound checkpoint, runtime INT8 target LM-head BF16 scales, runtime INT4 draft LM-head BF16 scales, ReplaySSM exact GDN state handling, target-verified MTP3, unique cold prompts, and cached_tokens=0. The pinned public oneCCL/libccl revision fixes deterministic packed-verifier all-reduce graph failures in the installed runtime; an opaque compiled all-gather boundary then enables exact draft graph capture. Both swapped four-GPU crossover assignments favored the transaction candidate. See the TP2 record packet. LocalMaxxing approved the 95.384868 tok/s row as cmrh35ct50092mj01h7jgydqj; the prior 93.036242 tok/s row is cmrgue7kl007pmj01yrkcyqmv. The previous one-B70 record was 68.236 tok/s (LocalMaxxing cmr9atqb800msqr01u760xh0t), and the previous BF16-scale INT8-LM-head-only record was 65.276 tok/s. The service/prompt-processing lane is separate: see ../experiments/qwen36-27b-autoround-int4-b70/notes/2026-07-04-long-context-ladder-baseline.md for the cold long-context ladder, including the 32K-capability anchor through 17706 actual prompt tokens with exact JSON retrieval and cached_tokens=0.

Gemma 4 26B Short Decode And Service Lanes

Start with the handoff and production backend recipe: ../results/gemma4-26b-a4b-q8-b70/HANDOFF.md and ../results/gemma4-26b-a4b-q8-b70/production-service.md. For the temporary OpenAI-compatible Gemma 26B endpoint used by coding agents, start from gemma4-26b-q8-service-runbook.md; it covers both the validated four-GPU 8 x 64K setup and a one-GPU 2 x 64K setup.

The Gemma 4 26B one-B70 settings are documented in ../results/gemma4-26b-a4b-q8-b70/reproduce.md. The standalone strict short-decode repro is ../repro/gemma4-26b-a4b-q8-b70-125tps-20260701/. The older ../repro/gemma4-26b-a4b-q8-b70-95tps-20260624/ folder remains useful as a standalone prior recipe, but it is superseded by the 124.97714084813418 tok/s fixed realistic-suite record in the Gemma result packet. The service/prompt-processing baseline is separate from that short record: GGML_SYCL_FATTN_DV512_GQA_NCOLS2=8, SWA left-bound, KQ register/broadcast, phase prefill ubatch 2048, 32K context, FA on, and VMM. The 2026-07-02 service ladder passed all 32 long-context rows with exact JSON validation and cached_tokens=0; average lane median prefill was 1192.965 tok/s, average lane median long-context decode was 131.786 tok/s, and the longest 32571 actual-token prompt stayed around 991-1006 tok/s prefill and 114-115 tok/s decode. Start with ../repro/gemma4-26b-a4b-q8-b70/run-vdr2-long-context-service-gate.sh, ../experiments/gemma4-26b-a4b-q8-b70/sweeps/20260702-current-service-context-ladder.md, and ../data/gemma4-long-context-service-gate-20260702Tservice-ladder-current-rep4.json. Older prefill diagnostics, including KV-max scan, forced ncols1, nbatch_fa=128, and phase-ubatch screens, remain archived under ../experiments/gemma4-26b-a4b-q8-b70/sweeps/.

Writing Future Docs

When adding or refreshing docs for another model:

This site is open source. Improve this page.