b70-optimization-lab

Qwen3.6 Decode Graph Replay Corruption Boundary

Date: 2026-06-13

Summary

The fast TP4 PIECEWISE graph lane for nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 is not quality-safe yet. The failure is below the chat wrapper and appears in raw /v1/completions with a fixed rendered ChatML prompt.

The decisive new boundary:

This means the next speed work should repair decode replay state/lifetime correctness, not tune output copying or response formatting.

New Probe

Added:

It sends raw rendered ChatML to /v1/completions and repeats deterministic fixed prompts. It now supports --logprobs and records HTTP errors as mismatches, which caught the logprob/NaN failure below.

Compile check:

/home/steve/.venvs/vllm-xpu/bin/python -m py_compile \
  scripts/probe-fixed-chatml-completion-repeat.py \
  scripts/qwen36-text-quality-suite.py

Results

Accepted Fast Graph Corruption

Known fast controls:

Representative failures:

Clean No-Replay Control

Global no-replay control:

Selective decode-replay bypass:

MAX_NUM_SEQS=1 \
MAX_NUM_BATCHED_TOKENS=8192 \
LOG_PATH=/tmp/qwen36-quark-int8-tp4-seq1-nodecodereplay-probe-20260613o.log \
VLLM_XPU_DISABLE_DECODE_CUDAGRAPH_REPLAY=1 \
scripts/launch-qwen36-quark-int8-accepted.sh

Artifacts:

Interpretation: this is a correctness control and a replay-boundary proof, not a promoted speed candidate.

Negative Fix Attempts

VLLM_XPU_SYNC_ASYNC_OUTPUT_COPY=1:

MAX_NUM_SEQS=1 \
MAX_NUM_BATCHED_TOKENS=8192 \
LOG_PATH=/tmp/qwen36-quark-int8-tp4-seq1-syncoutput-probe-20260613o.log \
VLLM_XPU_SYNC_ASYNC_OUTPUT_COPY=1 \
scripts/launch-qwen36-quark-int8-accepted.sh

VLLM_XPU_CUDAGRAPH_MARK_STEP_BEGIN=1:

Runtime recapture:

Strong output:

Logprob/NaN Evidence

The logprob probe on the fast replay path failed with a server-side JSON serialization error:

The server log also recorded the same ValueError while rendering logprobs. This points at corrupted logits/logprob tensors, not merely detokenization, HTTP response handling, or host output-copy ordering.

External Signals

These were not direct Intel-B70 fixes, but they match the failure class and informed the next direction:

cudagraph_mark_step_begin() did not fix our vLLM/XPU manual graph replay failure, but the stale-state and grouped-kernel graph warnings remain relevant.

Focused Patch Snippets

Saved focused patch:

This patch is diagnostic-only:

Only the decode replay bypass produced clean output, and it is too slow to promote.

Next Best Work

  1. Add a first-divergence replay-vs-no-replay tensor/logits microscope for the fixed color prompt. Capture token ids, positions, slot mappings, block table slice, hidden-state digest, sample-hidden digest, logits finite counts, and top-k logits around the first bad generated token.
  2. Inspect all graph-replayed modules for Python-side state writes during forward. The PyTorch stale-state issue makes this a plausible source of replay-only wrongness.
  3. Add an opt-in static-input sanitation mode before replay: fill unused/padded regions of input ids, positions, slot mappings, and block tables with sentinels or zeroes, then rerun the fixed canary.
  4. Add graph-wrapper replay counters by subgraph/layer and correlate the first fixed-prompt mismatch with the replay counts for MoE, GDN, attention, and logits.
  5. If tensor/logit microscope points at one layer family, try a selective replay bypass only for that family instead of all decode. The target is to recover most of the ~100 tok/s graph speed while keeping the clean canary behavior of the 15.99 tok/s control.
  6. Keep persistent W8A8 MoE layerlet work alive, but do not stack more MoE speed claims on top of a corrupt graph replay base.

2026-06-13q Follow-Up: Replay Boundary Repairs Tried

New probe support:

Key trace finding:

Fast-but-not-clean candidate:

Repairs tested and rejected:

Current conclusion:

Next best ideas:

  1. Selectively bypass graph replay for GDN/Mamba subgraphs only, if the piecewise runner can route that family eager while keeping MoE/dense replay.
  2. Add a layer-family replay microscope or state fingerprint around GDN cache writes: record state indices, has_initial_state, conv/SSM state norms for fresh prefill blocks, and per-layer GDN output digests.
  3. Investigate the XPU graph/noop communicator capture path. A proper XPU communicator graph-capture context may be required if noop capture lets replayed collectives interact with stale external state.
  4. Test static input and metadata sanitation before every PIECEWISE replay: zero unused/padded request rows, block-table rows, logits indices, and metadata tensors before copying active data.
  5. If selective family bypass is not practical, implement a low-overhead periodic replay quarantine that disables replay for an entire request after a fixed number of successful graph-replayed requests, then use canaries to find whether corruption is count-based or request-content-based.

2026-06-13y/ad Follow-Up: No-Async Boundary Isolation

New positive control:

Interpretation:

Native GDN zeroing update:

No-async PIECEWISE observations:

Repairs tested in this follow-up:

Current best correctness state:

Next deeper repair targets:

  1. Add replay call-site/family counters and a per-request graph replay trace to identify which PIECEWISE subgraph is replayed immediately before the color branch flip.
  2. Try selective family-level replay bypass, starting with GDN/Mamba and then logits/LM-head, instead of request-level first-N decode heuristics.
  3. Repair or replace the cudagraph_copy_inputs=true path for XPU. The stock path matches the suspected stale-input class but fails at startup with an inductor bounds assertion.
  4. Add static metadata sanitation before replay for positions, logits indices, slot mappings, and block tables. This is lower overhead than global graph disablement and directly targets stale/padded replay inputs.
  5. Keep the graph-disabled lane as the only quality-safe fallback until both JSON and color pass at 512 repeats under any faster replay candidate.

Replay Boundary Follow-Up

Patch snapshot:

Added opt-in replay tracing in vllm/compilation/cuda_graph.py:

Artifacts:

Findings:

Selective hybrid replay bypass was not a valid repair path yet:

Next action:

Replay Correctness Follow-Up 2026-06-13am-ar

Patch snapshot:

New instrumentation and knobs:

Artifacts:

Findings:

Rejected candidates:

Current interpretation:

Next actions:

  1. Add a selective W8A8 INT8 linear BF16 fallback diagnostic. If it survives replay-on JSON past repeat 53, the native int8_gemm_w8a8 path is the leading suspect.
  2. If the BF16 fallback is too large to fit globally, add a layer/submod-scoped fallback or trace around layer-0 dense projections first.
  3. Add replay-count-triggered direct-output digest around submod_0 with the normal replay-on path to confirm whether the corruption appears abruptly at a specific replay count.
  4. Keep VLLM_XPU_DISABLE_DECODE_CUDAGRAPH_REPLAY=1 as the correctness-safe fallback while searching for a replay-on repair.

INT8 Linear / MoE Fallback Follow-Up 2026-06-13as-av

Patch snapshot:

New diagnostics:

Artifacts:

Findings:

Updated interpretation:

Next actions:

  1. Inspect gdn_attn.py and gdn_linear_attn.py for persistent decode metadata, state-index tensors, temporary buffers, and XPU custom op calls that may be captured by graph replay.
  2. Add the smallest GDN-specific correctness knob: either force the GDN decode core to run outside replay, clone/zero its mutable state inputs, or disable a suspicious XPU fused path.
  3. Re-run the same JSON canary to check whether the repeat-53 failure moves or disappears.

GDN Native Fallback Follow-Up 2026-06-13aw-ay

Patch snapshot:

New diagnostic knob:

Artifacts:

Results:

Updated interpretation:

Next actions:

  1. Run replay output digest tracing for adjacent requests 65 and 66 with VLLM_XPU_GDN_NATIVE_FALLBACK=decode.
  2. If the first divergence is still before/at the first GDN boundary, add GDN fallback-branch state/output trace and compare selected conv/SSM state.
  3. If the first divergence moves to a later subgraph, target that graph family next rather than continuing to widen GDN fallbacks.

Sampler And Prefill Replay Repair 2026-06-14ba-bd

Patch snapshots:

New diagnostic/correctness knob:

Key trace finding:

Artifacts:

Results before the prefill replay guard:

Working correctness stack:

Validation artifacts:

Validation results:

Updated interpretation:

Next actions:

  1. Promote the working stack into a reproducible launcher or model-slot file so it is not lost.
  2. Run a small quality suite against the working stack and the BF16/eager fallback where practical, comparing token IDs on deterministic prompts.
  3. Measure the speed cost of VLLM_XPU_DISABLE_PREFILL_CUDAGRAPH_REPLAY=1. If decode speed is unaffected, keep it for correctness and move performance work back to decode/MoE.
  4. If prefill speed matters, localize the bad non-uniform prefill subgraph with VLLM_XPU_CUDAGRAPH_REPLAY_TRACE_SUBMOD_REGEX and replace the broad prefill bypass with a narrower submodule or batch-boundary guard.

Promotion:

Guard Cost Ablations 2026-06-14be-bj

New reproducibility helper:

Artifacts:

Results:

Updated interpretation:

Next actions:

  1. Keep the accepted launcher conservative until a replacement passes long repeat canaries after speed warmup.
  2. For sampler recovery, stop testing argmax-family variants unless we can route to a genuinely different XPU kernel; top-k is the only validated greedy reduction so far.
  3. For the next real speed win, target GDN native replay/state correctness or a narrower GDN fallback:
    • trace native GDN failures after several p512/o512 warm requests;
    • compare first failing JSON/color request against the GDN fallback stack;
    • inspect conv/SSM state indices and native op output at the first bad token.
  4. Continue larger decode work in parallel:
    • all-rank timing trace for the conservative stack;
    • persistent W8A8 MoE layerlet prototype;
    • TP/collective replay and topology work.

Native GDN Trace Follow-Up 2026-06-14bk-bn

Trace artifacts:

Reproduction profile:

Result:

Native GDN state finding:

Interpretation:

Rejected trace-derived fixes:

Current status:

Next actions:

  1. Compare the same traced request window against the accepted full decode GDN fallback stack to confirm whether fallback normalizes the selected layer-0 SSM state or merely tolerates the mismatch.
  2. Add a prefill-side GDN state trace for repeated identical prompts, not just decode-side trace, to determine whether the selected SSM mismatch is created during prefill or by decode metadata/state-index selection.
  3. If prefill creates the mismatch, inspect native prefill GDN state write coverage and has_initial_state handling.
  4. If decode metadata selects the wrong SSM slot, trace non_spec_state_indices_tensor, block IDs, and GDN state index mapping across requests 47/48.
  5. In parallel, shift performance work back to non-GDN paths that do not risk correctness: all-rank timing and W8A8 MoE layerlet.

W8A8 GDN Prefill Repair And Graph Gate 2026-06-14d1-d6

Problem:

Patch:

Artifacts:

Results:

Interpretation:

Promotion:

Next actions:

  1. Keep this no-graph lane as the correctness oracle for W8A8 XPU.
  2. Repair selective decode graph replay instead of broad no-graph execution:
    • compare first divergence between PIECEWISE and no-graph on the same canary window;
    • trace graph replay input/output digests around GDN state, MoE outputs, and collectives;
    • test copy/sanitize of only replayed static buffers that are reused across requests.
  3. Recover performance by re-enabling only proven-safe graph islands:
    • start with non-GDN/non-MoE subgraphs;
    • promote one island at a time through JSON/color canaries and quality suite;
    • reject any island that fails after warm p512/o512 runs.
  4. Continue the larger no-quality-loss performance work:
    • all-rank decode timing with graph disabled as the correctness baseline;
    • persistent W8A8 MoE layerlet replay harness;
    • TP/collective replay and topology tests.

Prefix Replay And Submod0 Direct-Compare 2026-06-14e1-e4

Goal:

Patch:

Artifacts:

Results:

Interpretation:

Rejected:

Next:

  1. Stop treating PIECEWISE replay as an incremental promotion path until a deeper graph-runtime issue is found.
  2. Keep graph-disabled W8A8 as the correctness oracle.
  3. Move effort to one of two paths:
    • full direct decode optimization, especially persistent W8A8/GDN/MoE layerlets with explicit owned buffers;
    • lower-level XPU graph/runtime debugging outside vLLM with a reduced submod0/submod1 reproducer.
  4. If graph replay is revisited, build the reduced reproducer first:
    • submod0 output tuple plus submod1 GDN consumer;
    • repeated replay/direct loops with token-ID canary state;
    • explicit checks for allocator/pool reuse, stream ordering, and custom-op mutation.

2026-06-14 Native-Decode / Safe-Prefill Follow-Up

Goal:

Code changes:

Artifacts:

Results:

Interpretation:

Rejected:

Accepted as oracles only:

Next:

  1. Treat VLLM_XPU_FORCE_GRAPH_WITH_COMM=1 plus VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1 as the corruption boundary.
  2. Build a reduced forced-comm graph replay reproducer around the first graph island and its collective/scratch behavior, rather than continuing endpoint A/B guesses.
  3. Add or reuse trace hooks that can compare state before and after replay across requests, including allocator addresses, graph-owned output buffers, collective wait tensors, and any persistent scratch tensors.
  4. In parallel, keep the graph-disabled lane as the correctness oracle while optimizing direct decode only if graph-runtime repair stalls.

2026-06-14g Forced-Graph Recapture Diagnostic

Goal:

Patch:

Results:

Interpretation:

Rejected:

Next:

  1. Add trace counters for actual recapture events by wrapper label to confirm whether piecewise:0/ selective recapture is unexpectedly triggering slow graph-stack behavior.
  2. Replace runtime recapture with cheaper state refresh:
    • identify graph-owned/static input buffers for piecewise:0/;
    • copy/sanitize only those buffers before replay;
    • avoid destroying/rebuilding the graph.
  3. Revisit cudagraph_copy_inputs=true failure with a minimal XPU-safe copy implementation instead of the stock path that hit an inductor bounds assert.
  4. Add a launch-cooldown/retry guard for repeated XPU benchmark runs because the device-lost failure appeared during startup after several heavy runs.

2026-06-14h Run-Config Drift Correction

Problem found:

Harness fix:

Harness verification:

Corrected fast baseline:

Corrected safety result:

Rejected corrected candidates:

Current status:

Next:

  1. Stop broad endpoint A/B until each candidate uses the harness autoconfig or explicitly records why PIECEWISE is disabled.
  2. Build the reduced forced-comm PIECEWISE replay reproducer around request boundaries, not runtime recapture.
  3. Trace static graph input/output buffer identities, allocator addresses, and collective scratch tensors across good and bad requests.
  4. Test a targeted static-buffer refresh/copy-input patch for the first request-boundary graph island, then gate it on:
    • restored fast config;
    • JSON/color canaries at 128+ repeats;
    • p512/o512 speed against the 20260614h4 baseline.

2026-06-14i Graph-0 Output Alias Diagnostic

Goal:

Patch:

Trace artifacts:

Trace findings:

Repair attempts:

Interpretation:

Rejected:

Current status:

Next:

  1. Run the same graph-output trace after the metrics warmup sequence used by h15/h16, targeting the failing JSON repeat 48 and color repeat 91. The clean-only h12 trace is no longer sufficient.
  2. Add an alias-lifetime trace that directly logs whether graph0 output 5 shares storage or data pointers with graph1 tensor arguments or downstream mutable scratch buffers.
  3. Replace manual index cloning with a principled handoff rule: clone or allocate an owned per-replay buffer only when a graph output is later used as a mutable input by the next graph island.
  4. Only after the metrics-warmed canaries pass, re-run:
    • at least four p512/o512 measured repeats;
    • JSON/color canaries at 128+ repeats after metrics;
    • quality suite against the graph-disabled oracle.

2026-06-14j Metrics-Warmed GDN State Isolation

Why:

Runs:

Current accepted code state:

Decision:

Next:

  1. Add targeted Python fallback GDN prefill trace for only adjacent good/bad requests around JSON 46-49 and color 135-138.
  2. Digest these tensors at graph0/layer0 prefill: projected_states_qkvz, projected_states_ba, mixed_qkv_non_spec, conv_state at selected state indices, initial_state, core_attn_out_non_spec, and last_recurrent_state.
  3. If the divergence first appears before GDN core, inspect projection/input quant static buffers.
  4. If the divergence first appears inside GDN core with identical inputs/state, isolate the recurrent fallback kernel.
  5. If GDN internals remain identical but graph0 output differs, instrument the post-GDN norm/output projection handoff.

2026-06-14k Python GDN Fallback Trace

Why:

Instrumentation:

h26:

h26 finding:

h27:

h28:

Current conclusion:

2026-06-14l Suggestions.md Triage And Cleanup

Source:

Accepted direction:

Next work to try, in order:

  1. Metadata/static replay input sanitation before XPU graph replay.
    • Gate with a new env such as VLLM_XPU_CUDAGRAPH_SANITIZE_METADATA_TAIL=1.
    • Before replay, zero or sentinel-fill inactive tails for positions, slot mappings, block tables, logits indices, and attention metadata.
    • Keep this narrow; do not copy all tensors unless the targeted sanitation proves insufficient.
    • Pass condition: restored fast-lane canaries survive after metrics warmup without decode speed collapsing.
  2. Repair an XPU-safe copy-input graph path.
    • Reproduce the prior cudagraph_copy_inputs=true inductor bounds failure with logs and generated-kernel context.
    • Add a minimal XPU-safe copy-input mode if the failure is a workspace or active-region bounds issue.
    • Gate separately, for example VLLM_XPU_CUDAGRAPH_COPY_INPUTS_SAFE=1.
  3. If graph replay remains unsafe, move performance effort to clean direct decode.
    • Prototype persistent W8A8 MoE layerlet kernels using the existing scratchpad-ring work as the base.
    • Keep weights, expert pointers, scales, route buffers, scratch, and output buffers resident.
    • Validate against the graph-disabled or clean 85-86 tok/s oracle before any speed claim.
  4. Profile communication and parallelism after correctness is stable.
    • Run collective replay for current TP4, reversed rank map, and TP2.
    • Test PP=2/TP=2 only if communication is a measurable single-request wall.
    • Track aggregate throughput separately from single-request decode.

Rejected next steps:

Disk cleanup:

Immediate implementation target:

2026-06-14m Suggestions.md Second Review

Why this update exists:

Current baseline framing from the updated suggestions file:

Additional ideas and corrections to carry forward:

  1. Native GDN decode/prefill stability is now the nearest-term speed branch.
    • The updated suggestions file makes this priority #1 because native paths have already measured near or above 90 tok/s.
    • Required next gate: promote the native-decode-safe-prefill-graph lane through full quality, longer JSON/color canaries, and repeated p512/o512.
    • If it fails warm validation, continue the stale graph-pool / allocator investigation at the upstream projection/custom-op boundary rather than adding more GDN state zeroing.
  2. W8A8 MoE layerlet + prefix offsets should remain a top implementation bet.
    • The code and harness are already partially present: csrc/xpu/moe_layerlet.cpp, grouped-GEMM offset interfaces, and xpu_moe.py workspace reuse.
    • Current failure mode from suggestions: about 72 tok/s and immediate color canary failure, likely offset or scale alignment rather than a fundamental kernel problem.
    • Next debugging target: trace the first divergent tensor versus reference under VLLM_XPU_W8A8_EXPERIMENTAL_ALLOW=1, then fix offset computation or scale broadcasting before testing VLLM_XPU_W8A8_USE_OFFSETS=1, VLLM_XPU_W8A8_OFFSETS_PREFIX_OP=1, and VLLM_XPU_MOE_W8A8_MIDDLE_LAYERLET=1 together.
    • Only combine VLLM_XPU_INT8_MOE_MIXED_WORKSPACE=1 after quality is clean.
  3. Replace no-op communication capture with real XPU graph collectives.
    • VLLM_XPU_GRAPH_NOOP_COMM_CAPTURE=1 is still a workaround.
    • A real XPU graph-captured custom all-reduce could remove host dispatch and sync overhead, and may reduce graph/collective state leaks.
    • This is larger work but has medium-high upside.
  4. Graph replay overhead cleanup is a performance item, not only a correctness item.
    • The metadata-sanitation experiment reportedly cost roughly 8 tok/s, proving replay input preparation and metadata copies are material.
    • Follow-up targets: block-table updates, redundant host-to-device copies in cuda_graph.py, metadata tensor reuse, and per-replay event/stream creation.
  5. Custom all-reduce clone removal is a possible win but must be treated as dangerous.
    • Existing notes already say the inner clone was required for earlier safe custom all-reduce lanes.
    • Revisit only under the exact newer native-prefill graph identity and with full gates. Test VLLM_XPU_CUSTOM_ALLREDUCE_CLONE_INPUT=0 and VLLM_XPU_CUSTOM_ALLREDUCE_GRAPH_CLONE_INPUT=0 separately, not together first.
  6. Keep a broader software-only backlog:
    • TP4 vs PP2/TP2 or PP4 reshape if collectives remain hot.
    • Kernel fusion for RMSNorm/residual, activation+quant, RoPE+QKV, and SiLU/mul/quant.
    • Causal-conv1d / GDN Xe2 tile and scratch tuning.
    • Runtime autotuning of PIECEWISE split points.
    • XPU/Inductor compilation config search, but avoid async scheduling until quality evidence changes.
    • Weight layout / pre-packed oneDNN or Xe2-blocked INT8 layouts.
    • Engine-core scheduler overhead reduction for batch-1 decode.
    • Persistent MoE/attention kernels and compute/communication overlap.

Updated near-term priority:

  1. Full quality promotion attempt for the 93.3 tok/s native-decode-safe-prefill-graph lane.
  2. If the lane fails, trace the first warmed divergence at the graph replay / projection boundary with the lightest possible digest.
  3. Fix W8A8 MoE layerlet prefix-offset correctness, because it attacks the largest remaining INT8 MoE overhead without reducing quality.
  4. Profile and then work real XPU graph collectives / all-reduce overlap if TP synchronization remains a measured wall.
  5. Do graph replay metadata-copy cleanup after the correctness lane is stable, so speed wins are not confused with corruption workarounds.
This site is open source. Improve this page.