b70-optimization-lab

Qwen 3.6 35B INT8 XPU Optimization Handoff

Date: 2026-06-16

Current Goal

Get nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 on 4x Intel Arc Pro B70 well above 100 tok/s single-request decode with no quality loss.

Quality loss is not acceptable. Any promoted change must be token/canary/quality validated against the current safe fast identity before being treated as a win.

Benchmark Identity Rule

Do not compare runs unless the full run identity matches.

Critical identity fields:

Previous mistake to avoid: a run missing COMPILATION_CONFIG='{"cudagraph_mode":"PIECEWISE"}' defaulted to graph-none and measured around 15 tok/s, which was incorrectly compared against the fast PIECEWISE forced-comm lane.

Known fast-but-unsafe historical baseline: PIECEWISE forced-comm graph lane at about 93.45 tok/s.

Current validated safe research base:

Current Safe Fast Identity

Model:

/mnt/fast-ai/llm-cache/hf/models--nameistoken--Qwen3.6-35B-A3B-Quark-W8A8-INT8/snapshots/cced56592e8c8935f8220836b4baa04dfd389118

Core settings:

COMPILATION_CONFIG='{"cudagraph_mode":"PIECEWISE"}'
XPU_GRAPH=1
VLLM_XPU_ENABLE_XPU_GRAPH=1
VLLM_XPU_FORCE_GRAPH_WITH_COMM=1
VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1
VLLM_XPU_GDN_NATIVE_FALLBACK=prefill
VLLM_XPU_GDN_PREFILL_RECURRENT_FALLBACK=1
VLLM_XPU_DISABLE_PREFILL_CUDAGRAPH_REPLAY=1
VLLM_XPU_GREEDY_SAMPLE_TOPK_FALLBACK=1
VLLM_XPU_INT8_MOE_MIXED_WORKSPACE=1
GPU_MEMORY_UTILIZATION=0.90
VLLM_EXTRA_ARGS='--uvicorn-log-level warning'

Where I Am Right Now

2026-06-19 update: the latest decisive no-async TP2 PIECEWISE trace changed the immediate priority. MoE is still the largest model-forward family, but the accepted no-async decode path is also paying a large sampled-token materialization cost after sampling.

Diagnostic identity:

TP_SIZE=2
ONEAPI_DEVICE_SELECTOR=level_zero:0,1
COMPILATION_CONFIG='{"cudagraph_mode":"PIECEWISE"}'
XPU_GRAPH=1
VLLM_XPU_ENABLE_XPU_GRAPH=1
VLLM_XPU_FORCE_GRAPH_WITH_COMM=1
VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1
VLLM_XPU_GDN_NATIVE_FALLBACK=prefill
VLLM_EXTRA_ARGS='--no-async-scheduling --uvicorn-log-level warning'

New diagnostic artifacts:

Key finding:

Implication:

The next high-upside exact path is output materialization overlap/removal, especially repairing async scheduling correctness or building a narrower deferred sampled-token output path. Exact MoE/shared-expert work remains important, but it cannot by itself reach the target while _to_list costs about 4.6 ms/token in the safe no-async path.

Async isolation results:

Follow-up async/PIECEWISE replay isolation:

Interpretation:

Older context: before this trace, the work was focused on the fused W8A8 MoE prologue path, because earlier family timing showed MoE/shared-expert work dominating the single-request decode budget.

Important timing artifact:

Timing conclusion from that artifact:

So the current high-upside branch is still MoE/shared-expert structural work, not TP topology or sampler-only work.

Latest Fused-Prologue Findings

I added a layer skip/include gate for fused prologue capture so we can bisect which layers poison PIECEWISE graph capture.

Code changed:

New env vars:

VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET_SKIP_LAYERS
VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET_LAYER_REGEX

Syntax checks passed before the latest endpoint tests:

/home/steve/.venvs/vllm-xpu/bin/python -m py_compile /home/steve/src/vllm-xpu-kernels/vllm_xpu_kernels/fused_moe_interface.py
/home/steve/.venvs/vllm-xpu/bin/python -m py_compile /home/steve/src/vllm/vllm/model_executor/layers/quantization/quark/quark_moe.py

Latest endpoint readiness tests:

  1. skip0
    • Label: prefill-safe-int8-prologue-skip0-c1-ready
    • Log: /home/steve/llm-optimizations/data/qwen36-ablation-prefill-safe-int8-prologue-skip0-c1-ready-20260616prolskip0ready1.log
    • Result: failed before readiness with UR_RESULT_ERROR_DEVICE_LOST
    • Failure site: /home/steve/src/vllm/vllm/v1/worker/gpu_model_runner.py:9456 copying logit_indices_device = torch.from_numpy(logit_indices).to(...) during c1 PIECEWISE graph capture.
  2. skip0-3
    • Label: prefill-safe-int8-prologue-skip0-3-c1-ready
    • Log: /home/steve/llm-optimizations/data/qwen36-ablation-prefill-safe-int8-prologue-skip0-3-c1-ready-20260616prolskip03ready1.log
    • Result: failed before readiness with the same UR_RESULT_ERROR_DEVICE_LOST at the same dummy logit_indices copy during c1 graph capture.
    • Failed run left workers alive; I killed those exact orphaned PIDs. A later xpu-smi ps showed no vLLM workers left.
  3. skip0-39
    • Label: prefill-safe-int8-prologue-skipall-c1-ready
    • Summary: /home/steve/llm-optimizations/data/qwen36-ablation-prefill-safe-int8-prologue-skipall-c1-ready-summary-20260616prolskipallready1.json
    • Log: /home/steve/llm-optimizations/data/qwen36-ablation-prefill-safe-int8-prologue-skipall-c1-ready-20260616prolskipallready1.log
    • Result: passed readiness-only run.
    • Meaning: the layer gate works, and the surrounding W8A8/middle-layerlet flags are not enough by themselves to crash capture.
  4. skip0-19
    • Label: prefill-safe-int8-prologue-skip0-19-c1-ready
    • Log: /home/steve/llm-optimizations/data/qwen36-ablation-prefill-safe-int8-prologue-skip0-19-c1-ready-20260616prolskip019ready1.log
    • Result: failed with UR_RESULT_ERROR_DEVICE_LOST, then the API server timed out waiting for engine core startup.
    • No summary file was emitted.
    • Latest xpu-smi ps after failure showed only xpu-smi, no vLLM workers.

Interpretation:

Reduced Reproducer State

Main reproducer:

Recently added features:

Passing reduced artifacts:

Meaning:

Files I Am Actively Using

Optimization notes and plans:

Runners and launchers:

Current reproducer:

vLLM Python code:

XPU kernel bridge and native code:

Important patch directory:

What I Intend To Do Next

Immediate next steps:

  1. Finish bisection of fused prologue capture safety.
    • Since skip0-19 failed and skip0-39 passed, next likely tests are ranges that keep progressively fewer high layers enabled: skip0-29, skip0-34, skip0-37, then single-layer enables if needed.
    • These are readiness-only tests first.
    • If a non-all skip range reaches readiness, promote it to metrics and canaries before making any speed claim.
  2. Add live ABI/context logging only if bisection is ambiguous.
    • Candidate envs: VLLM_XPU_MOE_LIVE_ABI_FILE=/home/steve/llm-optimizations/data/...jsonl and VLLM_XPU_MOE_LIVE_ABI_MAX_LINES=2000.
    • Use this only when necessary because it adds overhead.

2026-06-18 Update: Spec Parity And EAGLE Data Path

The current credible path to >150 tok/s single-session decode without quality loss is still trained EAGLE speculation. Other measured routes either failed quality gates or lost too much speed. The verifier/spec-state work has moved:

New EAGLE data export support was added in /home/steve/src/vllm/vllm/v1/worker/gpu_model_runner.py behind opt-in envs:

VLLM_XPU_EAGLE_DATA_DUMP_DIR=<dir>
VLLM_XPU_EAGLE_DATA_DUMP_RANK=0
VLLM_XPU_EAGLE_DATA_DUMP_MAX_STEPS=<n>
VLLM_XPU_EAGLE_DATA_DUMP_DTYPE=bfloat16
VLLM_XPU_EAGLE_DATA_DUMP_SINGLE_TOKEN_ONLY=1

Normal inference is unchanged when these envs are unset. Dump shards are stitched by:

/home/steve/llm-optimizations/scripts/build-qwen36-eagle-dataset-from-dump.py

Smoke results:

Current implementation direction:

  1. Use vLLM’s existing method:"eagle" path first, not EAGLE3. It wraps a plain Llama-style draft as EagleLlamaForCausalLM.
  2. Draft config should use hidden size 2048, vocab size 248320, and a tiny one-layer Llama decoder. vLLM’s simple EAGLE model consumes [token_embedding, target_hidden_state] through model.fc.
  3. Build a loadable smoke checkpoint first, then train it from the dumped target hidden states. Only after it loads and produces non-trivial acceptance should we scale data generation and run PIECEWISE TP4 speed gates.

2026-06-18 Update: EAGLE-1 Loader Smoke

Created a reusable smoke-checkpoint generator:

/home/steve/llm-optimizations/scripts/create-qwen36-eagle1-smoke-checkpoint.py

Generated draft:

/home/steve/llm-optimizations/data/qwen36-eagle1-smoke-draft-20260618a

Checkpoint properties:

Loader/runtime smoke:

Conclusion:

The local EAGLE-1 checkpoint format is loadable and executable on XPU. The next required step is a real trainer that writes the same checkpoint format from the Quark INT8 hidden-state dataset, then a TP2 acceptance smoke, then larger data export/training before any TP4 PIECEWISE performance gate.

  1. If every layer-enabled fused-prologue capture path fails, stop spending time on captured fused prologue and pivot to one of:
    • persistent MoE layerlet that avoids graph-captured prologue state
    • exact oracle k=1/k2 speculation parity repair
    • graph capture alias/lifetime tracing in cuda_graph.py
  2. Keep notes and patches current.
    • Save diffs for the reproducer update.
    • Save diffs for the fused-prologue layer gate.
    • Update /home/steve/suggestions.md and /home/steve/llm-optimizations/notes/2026-06-14-qwen36-recovery-implementation.md after each meaningful result.

Promotion rules:

Current Process State

After the latest failed skip0-19 run, I checked xpu-smi ps.

Observed state:

So the machine was clean at the time this file was written.

2026-06-17 Addendum — Fused-Prologue Bisection Concluded (Systemic)

Readiness-only test skip0-37 was run with the full safe fast identity plus fused-prologue capture enabled and SKIP_LAYERS=0-37 (only layers 38-39 allowed to use fused-prologue capture).

This is the SAME failure site as skip0, skip0-3, and skip0-19, even though only 1-2 layers (38-39) used fused-prologue capture.

Conclusion: the failure is SYSTEMIC to “fused-prologue capture enabled at all,” not per-layer MoE math. Enabling captured fused prologue on any layer perturbs the c1 PIECEWISE capture such that the dummy sampler torch.from_numpy(...).to(device) copy device-losts. Every layer-enabled fused-prologue capture path will fail at this same site.

Implication: the layer bisection is futile. Stop spending time on captured fused prologue. The failure site is in the sampler/logits dummy run and the piecewise capture boundary machinery, not in MoE kernel math, so the remaining options are the handoff step-3 pivots.

Post-run cleanup: the 4 VLLM::Worker_TP* processes orphaned after the device-lost were SIGKILLed; xpu-smi ps confirmed all GPUs idle and no vLLM processes remain.

Note: the ablation runner now records VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET_SKIP_LAYERS and VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET_LAYER_REGEX in both the log echo and the summary JSON, so future fused-prologue runs carry full identity (this was previously a gap).

2026-06-17 Addendum — Pivot to Shared-Experts: Both Levers Fail

After concluding fused-prologue capture is systemically dead, pivoted to shared-experts on the working safe PIECEWISE lane (per the moe.shared_experts.apply_no_overlap ~2.74 ms/step budget). Two implemented levers were tested against the safe identity (only the one env var changed, so identity matches the 93.55 base).

Lever A — VLLM_XPU_SHARED_EXPERTS_STREAM=1 (aux-stream overlap):

Lever B — VLLM_XPU_SHARED_EXPERT_FUSED_ACT_QUANT=1 (fused silu+mul+quant + raw int8_gemm_w8a8 for the shared down_proj):

Shared-expert per-step timing breakdown (from qwen36-shared-expert-internals-timing-timing-decision-20260615a20.json, summed across layers): gate_up_proj 0.725 ms, silu_and_mul 0.725 ms, down_proj 0.549 ms, expert_gate 0.373 ms, gate_mul 0.146 ms. Most of this is real memory-bound int8 GEMM work for single-token decode, so the only realistic reductions are overlap (blocked) or kernel fusion (currently buggy and slower).

Conclusion: on the captured PIECEWISE lane, the two main shared-expert speed levers are not viable as-is. Remaining untested lower-ceiling levers: VLLM_XPU_SHARED_EXPERT_ACT_WORKSPACE (buffer reuse, tiny) and VLLM_XPU_MOE_SHARED_ADD_ALLREDUCE_CUSTOM_OP.

2026-06-17 Addendum — Fused-Act-Quant Deep-Dive: Kernel Is Correct

User directed fixing the fused-act-quant kernel. Deep investigation (standalone op-level comparison on XPU) shows the fused activation kernel is NOT the bug:

So the color-canary failure is NOT a math/correctness bug in the fused kernel. The mismatch record shows it occurred at repeat index 6 (repeats 0-5 were consistent) with a divergent <think> token sequence — i.e. NON-DETERMINISM under PIECEWISE capture replay, not a systematic error. Most likely cause: the fused path calls the non-_out variant silu_and_mul_quant_int8_xpu, which allocates its output tensors fresh each call; under cudagraph capture those pool allocations can alias across replays, breaking determinism. The _out variant (silu_and_mul_quant_int8_xpu_out) with persistent buffers likely fixes determinism.

The speed deficit (91.85 vs 93.55, +0.20 ms) is because launch_silu_and_mul_quant_int8 (int8_quant.cpp:85) is a non-vectorized two-pass scalar kernel, slower than the vectorized silu_and_mul + per_token_quant_int8_xpu it replaces. Even with determinism fixed AND the kernel vectorized, the theoretical ceiling is small (it only merges the silu read-pass with the quant read-pass; the GEMM is unchanged) — well under the ~7% needed for >100 tok/s.

Net: fused-act-quant is architecturally low-ceiling on this lane. The recurring blocker across fused-prologue, stream-overlap, and now fused-act-quant is the XPU PIECEWISE capture constraint itself.

2026-06-17 Addendum — Attacking the Capture Constraint (Stream Overlap)

Pursued the capture constraint as the root blocker for shared-expert stream overlap. Root-caused the failure:

Fix added (vllm cuda_graph.py, env-gated VLLM_XPU_GRAPH_NO_EMPTY_CACHE): no-op torch.xpu.empty_cache during capture for XPU. This is a real bug fix (vLLM was patching the wrong function on XPU) and unblocks aux-stream graph capture generally; gated so the default lane is unchanged.

Result with VLLM_XPU_SHARED_EXPERTS_STREAM=1 + VLLM_XPU_GRAPH_NO_EMPTY_CACHE=1 (full safe identity, both envs changed):

Conclusion: the capture constraint was NOT the real blocker for stream overlap’s value. Even capture-unblocked, shared-experts aux-stream overlap is slower on 4x Arc Pro B70, because the separate stream does not provide real compute concurrency on this hardware — the fork/join/sync overhead is paid with nothing hidden. Stream overlap is closed as a speed lever.

Overall MoE/shared-expert lane outcome (all on the captured PIECEWISE lane): fused-prologue (capture-dead), stream-overlap (capture-fixable but no concurrency -> slower), fused-act-quant (kernel correct but slower + replay-nondeterministic). No per-pass MoE lever reaches the +7% needed for

100 tok/s. The capture constraint plus the lack of XPU multi-stream concurrency together cap the captured-lane approach.

2026-06-17 Addendum — N-Gram Spec Decode: Prompt-Dependent + Nondeterministic

Pivoted to speculative decoding (native MTP is dead: the Quark INT8 checkpoint has 0 mtp.* weights, confirmed in notes/2026-06-10-qwen36-local-argmax-and-mtp-rejected.md). Tested n-gram spec decode (method:ngram, k=5, prompt_lookup_min=2/max=5, {"cudagraph_mode":"PIECEWISE","max_cudagraph_capture_size":128}) through the standard canary+quality pipeline (runner made launcher-configurable via SERVER_LAUNCHER; identity otherwise matches the safe lane). The prior 2026-06-10 note reported 114.86 tok/s, but that was on a HIGH-REUSE prompt.

Result (label prefill-safe-int8-ngram5-cg128-c1):

Conclusion: n-gram spec decode is NOT a reliable path to >100 tok/s with no quality loss on the standard benchmark. It regresses speed on general prompts (low acceptance) and is nondeterministic (json canary).

Session Summary (2026-06-17)

Four speed levers exhausted with clear mechanisms:

  1. fused-prologue capture: systemic device-lost (dead).
  2. shared-experts stream overlap: capture-fixable (added VLLM_XPU_GRAPH_NO_EMPTY_CACHE) but no XPU stream concurrency -> -21%.
  3. fused-act-quant: fused kernel proven bit-exact correct; slower (non-vectorized) + replay-nondeterministic; low ceiling.
  4. n-gram spec decode: near-zero acceptance on general prompts -> slower; json-canary nondeterminism; only helps repetitive workloads.

Concrete artifacts produced (all uncommitted, ready to commit):

The 93.55 tok/s base appears near the practical ceiling for this model/hardware on the captured PIECEWISE lane. Reaching a reliable >100 tok/s with no quality loss likely requires either a real draft-model spec decoder with high diverse-prompt acceptance (EAGLE; MTP weights are absent from this checkpoint) or custom MoE int8 GEMMs faster than oneDNN.

2026-06-17 Addendum — MTP Spec Decode: Acceptance Works, Draft Too Heavy

Reframed goal: >150 tok/s single-session decode (prefer >200), no quality loss. Only a learned-draft spec decoder can give that multiplicative win. Native MTP was thought dead (Quark INT8 checkpoint has 0 mtp.* weights), but the official FP8 snapshot IS present locally (models--Qwen--Qwen3.6-35B-A3B-FP8/snapshots/95a72...), so a hybrid checkpoint (Quark INT8 target + official FP8 MTP head) was built via scripts/create-qwen36-quark-fp8-mtp-hybrid.py (output /mnt/fast-ai/qwen36-quark-int8-fp8-mtp-hybrid).

GOTCHA found and fixed: the ablation runner exports MODEL_PATH (Quark default) which the hybrid-mtp launcher’s ${MODEL_PATH:-hybrid} kept, so the first run served the QUARK path (random/uninitialized MTP head) -> 0% acceptance. Re-running with MODEL_PATH explicitly set to the hybrid made MTP load the real weights.

MTP acceptance (hybrid, real weights): GOOD. Per-position acceptance 62%-100% (avg ~62% on natural-chat, ~88% on repetitive), acceptance length ~1.6-1.9 at k=1. So a learned draft matched to this target IS achievable; the hybrid head generalizes acceptably across the INT8/FP8 hidden-state gap. (Label prefill-safe-int8-hybrid-mtp-k1-real-c1.)

BUT two blockers remain:

  1. SPEED (the MTP draft is too heavy). The MTP head is a FULL MoE layer (256 experts, mtp.layers.0.mlp.experts.*), nearly as expensive as a target forward. Measured at k=1 PIECEWISE cg128: 69.46 tok/s (decode 14.42 ms/tok) -> SLOWER than the 93.55 base. The extra draft token costs ~12.8 ms (~1x a target forward). A draft that costs ~1x the target cannot yield big speedups. (A graph-none run measured 19 tok/s - the accepted-launcher default is cudagraph_mode:NONE; always set PIECEWISE+cg128 for valid spec-decode speed.)

  2. CORRECTNESS (parity bug persists). At k=1 PIECEWISE both json- and color-canary FAILED (the verifier bonus-row GDN recurrent/conv state divergence from recovery-implementation.md is not fixed).

Conclusion: MTP cannot reach >150 here because its draft is a full MoE layer (too expensive). n-gram is free but 0% diverse acceptance. The architecture that delivers >150 on diverse prompts is a LIGHTWEIGHT learned draft (EAGLE: 1-2 thin layers, no MoE). vLLM has extract_hidden_states.py (EAGLE data prep) and the eagle proposer, but the EAGLE training repo is NOT present locally and no trained EAGLE draft for this model exists on disk. Reaching the >150 goal therefore requires training an EAGLE draft on the Quark INT8 target (extract hidden states, clone SafeAILab/EAGLE, train) - a multi-hour task with XPU-compat risk.

2026-06-18 Addendum — Restored Oracle Identity + Native Prefill State Failure

Important re-anchor:

Failed native-prefill bridge attempts:

Conclusion:

Postprocess experiment status:

2026-06-18 Addendum 2 — Gate Semantics Fixed, Native Verifier Still Unsafe

Serial correctness anchor:

Native verifier probes after the restored serial anchor:

Current interpretation:

Serial verifier speed check:

2026-06-18 Addendum 3 - EAGLE k5 Ceiling And Rollout Result

EAGLE speed ceiling:

Real EAGLE draft results:

Decision:

2026-06-18 Addendum 4 - TP4 Health Blocker And Traceability Fix

Cached PIECEWISE traceability fix:

Launcher health preflight:

Fused-prologue isolation attempts:

Hardware/runtime health finding:

How to re-check health:

PHYSICAL_DEVICES=0,1,2,3 XCCL_DEVICES=0,1,2,3 \
  /home/steve/llm-optimizations/scripts/check-qwen36-xpu-xccl-health.sh

Healthy-subset check:

PHYSICAL_DEVICES=0,1,2 XCCL_DEVICES=0,1,2 \
  /home/steve/llm-optimizations/scripts/check-qwen36-xpu-xccl-health.sh

Current decision:

2026-06-18 Addendum 5 - Healthy-Device W8A8 MoE Replay Floor

Because physical XPU 3 is still unhealthy, the following runs are offline single-device replay diagnostics only. They are not endpoint speed results and are not comparable to TP4 serving throughput. They use ZE_AFFINITY_MASK=0 and ONEAPI_DEVICE_SELECTOR=level_zero:0 to avoid the wedged card.

Middle-layerlet correctness recheck:

Full route-exact MoE replay, current binary:

Offset/active-offset GEMM replay:

Hot-packed expert replay:

Fused SiLU+quant replay:

Graph replay timing diagnostic:

Updated target decision:

2026-06-18 Addendum 6 - Full W8A8 Layerlet Staged, Exact, Not Promoted

2025.3 binary staging:

Full layerlet implementation:

Exact replay result:

Live Python branch smoke:

Next exact MoE work:

2026-06-18 Addendum 7 - Fused Quant1 Tried And Gated Off

Fused quant1 candidates:

Results:

Restored guarded default:

Next exact MoE target:

2026-06-18 Addendum 8 - Fast Gather And Unchecked Middle Rejected

Fast gather:

Unchecked-middle wrapper:

Current best exact offline MoE artifact:

2026-06-18 Addendum 9 - Real-Routing Oracle Fixed A Hidden Map/Binary Mismatch

Harness fixes:

Correctness bug found:

Build lesson:

Corrected artifacts:

Patch snapshots:

Corrected hot-skew sweep summary:

rows xpu fused us prealloc us offset us middle layerlet us full layerlet us max diff
1 392.893 255.456 243.029 211.361 228.176 0.0
2 319.131 199.030 200.844 182.169 166.017 0.0
4 342.726 224.809 218.783 188.799 181.233 0.0
8 322.517 200.850 200.811 177.496 169.169 0.0
16 317.837 204.380 194.935 173.069 166.498 0.0

Current interpretation:

Next exact/no-quality-loss performance targets:

2026-06-18 Addendum 10 - Active Offset, Q1, And Fast Gather Retested

Corrected active-offset sweep:

rows active offset us plain offset us middle layerlet us full layerlet us
1 200.102 198.386 168.792 187.122
2 200.772 199.823 167.921 163.722
4 195.026 197.106 167.212 164.606
8 200.824 199.621 171.210 163.241
16 201.019 197.892 167.576 160.979

Decision: keep active-offset as a diagnostic only. It is exact, but not faster enough to promote.

Corrected q1/fast-gather retests:

Current next target:

2026-06-18 Addendum 11 - Real Rows=1 Scan, Q1/Fast-Gather Rejected Again

Stable real-route baseline:

Stable real-route rows=1 summary:

path min us mean us median us p90 us max us <=160 us rows
xpu scratch 248.525 262.639 257.973 275.888 292.490 0/24
plain offset 189.216 203.241 200.009 212.507 233.532 0/24
active offset 189.769 204.650 202.847 213.597 226.699 0/24
middle layerlet 161.001 173.428 171.181 180.944 193.764 0/24
full layerlet 157.215 169.210 166.545 177.622 188.859 2/24

Decision: the corrected full layerlet is the best current exact MoE replay path, but it is not enough for endpoint promotion. It clears the 160 us gate on only 2 of 24 real routes and still has a 188.859 us tail.

Same-env confirmation after runtime identity patch:

Rejected real-route q1 and gather variants:

Hot-expert physical packing simulation:

Decision: physical expert packing is not a safe global speed lever from this fixture. Keep it as an idea for per-layer/per-rank adaptive layouts only if a future trace shows a stable route distribution with a smaller tail.

Runtime environment lesson:

Immediate command-list real-route check:

Decision: keep immediate command lists as a possible endpoint stability/tail lever to retest only after TP4 health is restored. It is quality-neutral in offline replay, but it is not a standalone rows=1 MoE win and should not be combined into accepted endpoint manifests without full quality and reliability gates.

W8A8 grouped GEMM m8 policy checks:

Artifacts and results:

policy artifact full layerlet min/mean/median/p90/max us <=160 rows decision
rebuilt default /home/steve/llm-optimizations/data/qwen36-int8-moe-layerlet-default-rebuilt-firstdecode-routes-row1-device0-20260618.json 154.045 / 163.868 / 159.998 / 175.859 / 203.502 12/24 current rebuilt control
generic m8 run 1 /home/steve/llm-optimizations/data/qwen36-int8-moe-layerlet-m8policy-firstdecode-routes-row1-stable-device0-20260618.json 155.153 / 163.253 / 160.339 / 172.203 / 179.175 9/24 promising tail, not repeatable
generic m8 run 2 /home/steve/llm-optimizations/data/qwen36-int8-moe-layerlet-m8policy-repeat-firstdecode-routes-row1-device0-20260618.json 154.458 / 170.817 / 170.447 / 178.564 / 195.076 2/24 rejected for repeatability
generic m8 + immediate /home/steve/llm-optimizations/data/qwen36-int8-moe-layerlet-m8-immediate-firstdecode-routes-row1-device0-20260618.json 154.731 / 167.267 / 164.970 / 174.000 / 212.484 6/24 rejected
m8 only when N<K /home/steve/llm-optimizations/data/qwen36-int8-moe-layerlet-m8-nltk-firstdecode-routes-row1-device0-20260618.json 155.290 / 175.271 / 173.277 / 192.108 / 202.766 3/24 rejected
m8 only when N>K /home/steve/llm-optimizations/data/qwen36-int8-moe-layerlet-m8-ngtk-firstdecode-routes-row1-device0-20260618.log crashed before JSON flush n/a rejected, unsafe

Decision: do not promote m8 policy. Generic m8 can reduce a single-run tail, but the improvement is not stable across repeats. GEMM1-only m8 is slower. GEMM2-only m8 segfaulted, although device 0 survived a post-crash XPU smoke test. Keep the m8 code path as a diagnostic only; do not set any m8 policy env in accepted endpoint launchers.

Next exact/no-quality-loss work:

2026-06-18 Addendum 5 - Offline EAGLE Acceptance Gate

Purpose:

New script:

Offline acceptance results:

checkpoint eval set artifact mean accepted decision
token-heavy EAGLE2 capped corpus2+pos data/qwen36-eagle2-tokenheavy-offline-accept-capped-20260618.json 1.640625 baseline
rollout5 EAGLE2 capped corpus2+pos data/qwen36-eagle2-rollout5-offline-accept-capped-20260618.json 1.9296875 best offline signal
token-focused rollout5 capped corpus2+pos data/qwen36-eagle2-tokenfocused-rollout5-offline-accept-capped-20260618i.json 1.828125 rejected
token-heavy EAGLE2 smoke held-out data/qwen36-eagle2-tokenheavy-offline-accept-smoke-20260618.json 1.5689655 baseline
rollout5 EAGLE2 smoke held-out data/qwen36-eagle2-rollout5-offline-accept-smoke-20260618.json 1.7413793 best offline signal
token-focused rollout5 smoke held-out data/qwen36-eagle2-tokenfocused-rollout5-offline-accept-smoke-20260618i.json 1.6724138 rejected

Rollout5 details:

Rejected token-focused fine-tune:

EAGLE data coverage status:

Next EAGLE work:

2026-06-18 Addendum 6 - m8 Guard

The shape-specific diagnostic VLLM_XPU_W8A8_GROUPED_GEMM_N_GT_K_POLICY=m8 previously segfaulted before JSON flush. The grouped-GEMM policy override now ignores m8 when gemm_n > gemm_k, both for generic and N>K overrides:

This guard is not a performance win and does not change the accepted endpoint identity. It exists to keep future diagnostic sweeps from selecting the unsafe GEMM2 shape. Rebuild/stage _xpu_C after this edit before running any policy sweeps that import vllm_xpu_kernels._xpu_C.

2026-06-18 Addendum 7 - Expanded EAGLE Data Collection And Rejected Fine-Tunes

Expanded TP2 hidden-state export:

Expanded-data offline acceptance controls:

checkpoint expanded mean accepted decision
token-heavy control 1.046875 weak
rollout5 control 1.132812 current control

The new expanded set is substantially harder than the old capped corpus2+pos gate where rollout5 gets 1.929688. This explains why endpoint acceptance is still far below the synthetic k5 ceiling.

Fine-tune attempts against expanded data:

checkpoint capped corpus2+pos smoke expanded decision
rollout5 control 1.929688 1.741379 1.132812 keep as best current control
expanded low-LR mixed 1.914062 1.741379 1.125000 reject
expanded-only 1.875000 not run 1.226562 reject; improves expanded but hurts established gate
expanded oversample4 1.882812 1.793103 1.156250 reject; improves smoke/expanded slightly but hurts capped gate

Artifacts:

Decision:

2026-06-18 Addendum 8 - Deeper EAGLE Draft Attempts

Offline eval identity correction:

Trainer changes:

Results:

checkpoint capped stride-4 smoke stride-1 expanded stride-4 decision
2-layer rollout5 control 1.929688 1.741379 1.132812 current control
3-layer repeat-init expanded2 1.921875 1.793103 1.132812 reject; capped regression
3-layer repeat-init old-only 1.898438 1.793103 1.117188 reject; capped regression
3-layer residual-init expanded2, base frozen 1.890625 1.758621 1.125000 reject; capped and expanded regression
3-layer residual-init old-only, base frozen 1.929688 1.741379 1.109375 reject; neutral on accepted gates, worse expanded

Decision:

Checkpoint-selection follow-up:

2026-06-19 Addendum - Direct GEMM2+Gather Epilogue Rejected

User asked to pursue exact MoE optimization first, with GEMM2+gather epilogue fusion before escalating to a persistent one-dispatch layerlet.

Implementation:

Smoke replay:

Multi-route replay:

Decision:

Next exact MoE work:

2026-06-19 Addendum - DPAS TopK8 GEMM2+Gather Prototype Rejected

Follow-up to the direct GEMM2+gather diagnostic above. The direct scalar kernel was exact but slow because it abandoned the optimized DPAS grouped-GEMM path. I prototyped a narrower DPAS-backed top-k8 GEMM2+gather path to test the best plausible version of “GEMM2 + gather epilogue first” before escalating to the persistent layerlet.

Implementation:

Build/import:

Smoke replay:

Multi-route replay:

N-tile replay follow-up:

Decision:

Updated next move:

2026-06-19 Addendum - Workspace Atomic Reuse Rejected

After rejecting GEMM2+gather epilogue variants, I tested one smaller fixed-overhead cut before escalating to a true persistent layerlet: remove the tiny per-GEMM atomic_buffer = at::empty({1}, int32) allocation inside the grouped GEMM offsets launcher when called from the full layerlet.

Implementation:

Build/import:

Route replay:

Decision:

2026-06-19 Addendum - Route-Known GEMM1 Diagnostic Rejected

After the GEMM2+gather and workspace-atomic branches missed the promotion gate, I tested the symmetric GEMM1-side question: whether the c1/top-k8 route-known shape can bypass the generic grouped-GEMM scheduler for GEMM1 and directly launch one DPAS workgroup family per routed expert row.

Implementation:

Build/import:

Route replay:

Timing:

Decision:

2026-06-19 Addendum - TP4 Health Still Blocked; Route-Class AOT Not Broad Enough

After closing route-known GEMM1, I refreshed the two decision gates that decide whether to keep working TP4 endpoint timing or route-class-specific kernels.

TP4 XPU/XCCL health:

Decision:

Route-class AOT/cache refresh:

Decision:

2026-06-19 Addendum - Full Layerlet Graph Replay Rejected

I tested whether graph replay of the current exact full MoE layerlet removes enough launch/dispatch overhead to make the existing boundary worth promoting.

Artifacts:

Run identity:

Normal full layerlet timing:

Graph replay timing:

Decision:

2026-06-19 Addendum - q1 + Route-GEMM1 Combo Is Exact But Unstable

Before starting a larger rewrite, I checked whether already-rejected exact subpaths stack into a stable win. The only interesting combo was VLLM_XPU_MOE_W8A8_FUSED_Q1=1 plus VLLM_XPU_MOE_W8A8_ROUTE_GEMM1=1.

Initial combo sweep:

Repeat gate:

Repeat timing:

Decision:

2026-06-19 Addendum - DFlash Draft Is Active But Native Verifier Still Fails

I tested the public DFlash draft model as a possible no-quality-loss route to verified multi-token decode, but only as a quality/parity smoke. This was not a speed promotion.

Draft model:

Kernel/runtime prerequisite fixed:

Parity runner/tooling:

Smoke run:

Result:

First mismatch:

Decision:

Follow-up DFlash recovery flag tests:

Decision:

Additional DFlash follow-up on 2026-06-19:

Updated decision:

Additional DFlash follow-up on 2026-06-19, later:

2026-06-19 Addendum - Full Layerlet Endpoint Smoke Is Exact But Small

I ran a same-identity TP2 endpoint smoke to check whether the exact C++ qwen36_moe_w8a8_full_layerlet route-replay gain survives real endpoint execution. This used devices 0,1 because physical device 3 is still not TP4-safe.

Shared identity:

Control artifact:

Full-layerlet artifact:

Decision:

2026-06-19 Addendum - TP2 Decisive Timing Reframes The MoE Target

I ran decisive live decode timing on the same TP2 PIECEWISE identity used by the full-layerlet smoke above.

Control timing artifacts:

Full-layerlet timing artifacts:

Key timing result:

Decision:

2026-06-19 Addendum - Shared-Expert Fused Act/Quant _out Is Not Graph-Safe

I tested a narrower shared-expert optimization that reuses preallocated activation-quantization outputs instead of letting silu_and_mul_quant_int8_xpu allocate fresh tensors. The endpoint flag stack was:

Microbench result:

Endpoint shallow smoke, TP2 PIECEWISE identity:

Deep canary under normal PIECEWISE decode replay:

Diagnostic replay bypass:

Strong-output replay mitigation:

Decision:

The fastest exact path is still shared-expert structural work, but promotion requires deep canaries under normal PIECEWISE replay, not only eager replay.

2026-06-19 Addendum - Async Scheduling Was A Separate Correctness Poison

I retested the current TP2 PIECEWISE forced/noop-comm graph lane after separating async scheduling from GDN/shared-expert graph replay issues.

Clean TP2/no-async identity:

Control results:

Async separation:

Shared-expert fused _out retest under the clean no-async identity:

Routed full-layerlet retest under the clean no-async identity:

Updated immediate plan:

  1. Keep the no-async TP2 identity as the current safe benchmark lane.
  2. Try stacking the small shared-expert _out candidate only after any larger candidate passes; it is not enough to chase alone.
  3. Restore TP4/device health next. TP2 clean speed in the low 80 tok/s range is too far from the >150 tok/s target without either healthy four-card execution or a no-quality-loss multi-token path.
  4. If TP4 remains blocked, build the offline shared-expert replay harness and pursue a larger fused/persistent shared-expert boundary instead of more routed-expert layerlet variants.

2026-06-19 Addendum - TP4 Still Blocked, Routed+Shared Stack Does Not Matter

Device/topology health:

Conclusion:

Stacked routed+shared candidate:

Decision:

2026-06-19 Addendum - Shared-Expert C++ Boundary Is Correct But Small

Implemented a larger exact shared-expert W8A8 boundary:

Build lesson:

Offline replay:

Endpoint canary, same clean TP2/no-async identity:

Endpoint metrics, same clean TP2/no-async identity:

Decision:

2026-06-19 Addendum - Shared Boundary Timing and Next Branch

Decisive timing after the shared-expert C++ boundary:

Interpretation:

Runner hygiene:

Next active plan:

  1. Recover a quality-safe fast lane before more endpoint speed claims.
    • Current no-async TP2 safe lane is only ~82-84 tok/s.
    • TP2 async shallow smoke can pass, but deeper notes show async canary corruption; do not promote async until repeated canaries and the quality suite pass in the exact identity.
  2. Resume oracle/spec parity repair as the largest no-quality-loss path to >150 tok/s.
    • First target: oracle k=1 must be token-identical to no-spec on the same no-async/PIECEWISE identity.
    • Trace token IDs, positions, slot IDs, KV block state, hidden-state digest, and top-k logits at the first divergence.
    • Only after parity, test DFlash/EAGLE/ngram acceptance or wider target-verified multi-token decode.
  3. Keep TP4 blocked until device 3 XPU/XCCL health is repaired externally. TP3 is incompatible with this model, and TP2 is the only healthy exact topology available in this shell.

2026-06-19 Addendum - Fused Shared Gate/Mul Increment

Implemented an opt-in C++/SYCL epilogue inside the existing shared-expert W8A8 boundary:

Implementation details:

Build/install:

JOBS=4 /home/steve/llm-optimizations/scripts/build-vllm-xpu-kernels-xpu-c-only.sh
cp /tmp/vllm-xpu-xpu-c-only-2025/vllm_xpu_kernels/_xpu_C.abi3.so \
  /home/steve/src/vllm-xpu-kernels/vllm_xpu_kernels/_xpu_C.abi3.so

Import sanity:

PYTHONPATH=/home/steve/src/vllm:/home/steve/src/vllm-xpu-kernels \
LD_LIBRARY_PATH=/home/steve/src/vllm-xpu-kernels/vllm_xpu_kernels:/home/steve/.venvs/vllm-xpu/lib:/home/steve/.venvs/vllm-xpu/lib/python3.12/site-packages/torch/lib \
/home/steve/.venvs/vllm-xpu/bin/python -c 'import torch, vllm_xpu_kernels._xpu_C; print(hasattr(torch.ops._xpu_C, "qwen36_shared_expert_w8a8_out"))'

Validation identity:

TP_SIZE=2
ONEAPI_DEVICE_SELECTOR=level_zero:0,1
ZE_AFFINITY_MASK=0,1
QWEN36_XPU_PREFLIGHT=0
COMPILATION_CONFIG='{"cudagraph_mode":"PIECEWISE"}'
XPU_GRAPH=1
VLLM_XPU_ENABLE_XPU_GRAPH=1
VLLM_XPU_FORCE_GRAPH_WITH_COMM=1
VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1
VLLM_XPU_GDN_NATIVE_FALLBACK=prefill
VLLM_EXTRA_ARGS='--no-async-scheduling --uvicorn-log-level warning'
VLLM_XPU_SHARED_EXPERT_BOUNDARY_OUT=1
VLLM_XPU_SHARED_EXPERT_FUSED_GATE_MUL=1
GPU_MEMORY_UTILIZATION=0.90

Quality/canary artifacts:

Metrics artifact:

Comparable TP2/no-async controls:

Decision:

Next best branches after this increment:

  1. Routed MoE persistent one-dispatch layerlet: keep route buffers, descriptors, scales, scratch, GEMM workspaces, and output buffers resident; fuse route/remap, quant, GEMM1, activation/quant, GEMM2, and gather/combine. Gate offline first with exact replay parity.
  2. Exact speculation/oracle parity: repair k=1 token identity before attempting wider EAGLE/ngram acceptance. This remains the largest plausible no-quality-loss jump to >150 tok/s.
  3. Async/PIECEWISE replay correctness: graph-none async is correct, but async+PIECEWISE corrupts canaries. The N=8 eager workaround is correct but not faster. Treat this as a correctness branch, not a speed branch, unless a new replay ownership fix appears.
  4. TP4 only after hardware health: device 3/XCCL remains a blocker. Do not interpret TP4 endpoint speed until the preflight passes cleanly.

2026-06-19 Addendum - Real Full-Layerlet Endpoint Gate

Important correction:

Corrected endpoint identity tested:

TP_SIZE=2
ONEAPI_DEVICE_SELECTOR=level_zero:0,1
ZE_AFFINITY_MASK=0,1
QWEN36_XPU_PREFLIGHT=0
COMPILATION_CONFIG='{"cudagraph_mode":"PIECEWISE"}'
XPU_GRAPH=1
VLLM_XPU_ENABLE_XPU_GRAPH=1
VLLM_XPU_FORCE_GRAPH_WITH_COMM=1
VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1
VLLM_XPU_GDN_NATIVE_FALLBACK=prefill
VLLM_EXTRA_ARGS='--no-async-scheduling --uvicorn-log-level warning'
GPU_MEMORY_UTILIZATION=0.90
VLLM_XPU_W8A8_EXPERIMENTAL_ALLOW=1
VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET=1
VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET_ALLOW_CAPTURE=1
VLLM_XPU_MOE_W8A8_FULL_LAYERLET=1
VLLM_XPU_MOE_W8A8_FUSED_Q1=1
VLLM_XPU_MOE_W8A8_ROUTE_GEMM1=1
VLLM_XPU_SHARED_EXPERT_BOUNDARY_OUT=1
VLLM_XPU_SHARED_EXPERT_FUSED_GATE_MUL=1

Operational lesson:

Corrected run artifact:

Decision:

2026-06-19 Addendum - Sorted Active Helper And Q1 Direct Full-Layerlet Rejected

Current installed XPU extension state after the latest rebuild:

Sorted active-helper replay:

Q1 + direct GEMM2/gather full-layerlet:

Direct GEMM2/gather local-size 128 diagnostic:

Reviewed /home/steve/suggestions.md again:

Current conclusion:

2026-06-19 Addendum - oneDNN Sidecar Route Replay Rejected As Endpoint Path

Harness update:

Artifacts:

Seven-route identity:

Results:

Decision:

Next best branch:

2026-06-20 Addendum - Unchecked Full-Layerlet Validation Bypass Rejected

Diagnostic implementation:

Artifacts:

Identity:

Results:

Decision:

Next branch:

2026-06-20 Addendum - Route-GEMM1 B-Layout Fixed, Endpoint Gate Still Not Active

Root cause fixed:

Direct GEMM1 proof:

Replay results:

Endpoint A/B results:

Important correction:

Interrupted loose end:

Current active library hashes after trace instrumentation:

Saved patches:

Immediate next command to resume:

TRACE=/home/steve/llm-optimizations/data/qwen36-full-layerlet-trace-tp2-piecewise-gatereason-20260620.jsonl
GATE=/home/steve/llm-optimizations/data/qwen36-full-layerlet-gate-trace-tp2-piecewise-20260620.jsonl
rm -f "$TRACE" "$GATE"
STAMP=20260620tp2piecewisefulllayerletgatetrace2 \
PORT=18184 \
TP_SIZE=2 \
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 \
ZE_AFFINITY_MASK=0,1 \
QWEN36_XPU_PREFLIGHT=0 \
METRICS_REPEATS=1 \
METRICS_PROMPT_TOKENS=128 \
METRICS_OUTPUT_TOKENS=64 \
METRICS_WARMUP_OUTPUT_TOKENS=16 \
ABLATION_SKIP_CANARIES=1 \
ABLATION_RUN_QUALITY=0 \
ABLATION_SKIP_METRICS=0 \
VLLM_XPU_GDN_NATIVE_FALLBACK=prefill \
XPU_GRAPH=1 \
VLLM_XPU_ENABLE_XPU_GRAPH=1 \
VLLM_XPU_FORCE_GRAPH_WITH_COMM=1 \
VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1 \
COMPILATION_CONFIG='{"cudagraph_mode":"PIECEWISE"}' \
VLLM_XPU_W8A8_EXPERIMENTAL_ALLOW=1 \
VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET=1 \
VLLM_XPU_INT8_MOE_FUSED_PROLOGUE_OFFSET_ALLOW_CAPTURE=1 \
VLLM_XPU_MOE_W8A8_FULL_LAYERLET=1 \
VLLM_XPU_MOE_W8A8_FUSED_Q1=1 \
VLLM_XPU_MOE_W8A8_ROUTE_GEMM1=1 \
VLLM_XPU_MOE_W8A8_FULL_LAYERLET_TRACE_FILE="$TRACE" \
VLLM_XPU_MOE_W8A8_FULL_LAYERLET_TRACE_MAX_LINES=512 \
VLLM_XPU_MOE_W8A8_FULL_LAYERLET_GATE_TRACE_FILE="$GATE" \
VLLM_XPU_MOE_W8A8_FULL_LAYERLET_GATE_TRACE_MAX_LINES=512 \
/home/steve/llm-optimizations/scripts/run-qwen36-ablation-candidate.sh \
  tp2-piecewise-fulllayerlet-gatetrace-blayoutfix

Decision tree when resuming:

2026-06-21 Follow-Up - Full-Layerlet Gate Trace

The gate trace follow-up produced two useful artifacts:

The PIECEWISE trace repeatedly had no scratch keys or W8A8 offset scratch and never enabled the full-layerlet op (use_w8a8_full_layerlet=false). The graph-none mixed-workspace run had scratch keys, W8A8 offsets, and finally num_rows=1 with prologue_workspace=true and use_w8a8_full_layerlet=true.

Interpretation:

This site is open source. Improve this page.