b70-optimization-lab

Qwen3.6 27B Single-B70 Maximum-Speed Requirements And Execution Plan

Created: 2026-07-12

Status: controlling plan for the active Qwen27 TP1 optimization lane

Mandate

Maximize Qwen3.6 27B decode throughput on one Intel Arc Pro B70. The other three B70s are experiment accelerators for independent TP1 work, controls, and crossover validation. They are not a reason to move the target to TP2 or TP4.

The intended result is not a collection of small benchmark wins. The target is an unusually fast, quality-valid, reproducible single-card decoder built around maximum useful fusion, a replayed native device pipeline, and the fastest speculation policy supported by measured end-to-end economics.

Do not begin TP2 or TP4 optimization for this model until the user explicitly changes this mandate.

Non-Negotiable Requirements

  1. TP1 on one B70 is the product and headline configuration.
  2. Use all four B70s as independent TP1 workers to shorten iteration time.
  3. Pursue changes with a credible path to a large end-to-end gain. Close sub-percent and config-only lanes quickly.
  4. Build toward the fewest practical device submissions and global-memory intermediates, while preserving fast projection kernels and occupancy.
  5. Speculation must accept tokens verified by the declared target model.
  6. Prompt, KV, response, n-gram/history, and context reuse are forbidden for promoted fresh-response results.
  7. Model weights, B70-native weight packs, compiled kernels, representative activations, and diagnostic state snapshots may be cached and kept resident to accelerate development.
  8. Persist every meaningful patch, exact run identity, result, failure, and decision. Do not silently discard negative work.
  9. Do not promote a result until exact cases, repeat quality, recurrent/KV state behavior, cold realistic prompts, and cached_tokens=0 pass.
  10. No LocalMaxxing submission unless a strict matching record is genuinely broken and all submission-policy fields are present.

Model And Runtime Identity

Primary research identity:

Every comparison must record at least:

Current Evidence And Protected Work

As of plan creation:

Performance Model And North Star

Current approximate no-spec target pass:

observed pass             37.6 ms
estimated Q4 read floor   24.3 ms
perfect overhead removal  1.55x

Therefore graph replay, host cleanup, and fusion cannot create an incredible result alone at the current target byte count. They are required because they shorten every speculative cycle, but the large multiplier must come from one or both of:

  1. accepting several output tokens per target weight stream;
  2. materially reducing target bytes per weight stream.

Primary speed objectives:

Work Selection And Kill Rules

A development lane must satisfy at least one condition:

Otherwise give it one bounded microbenchmark and, at most, one paired endpoint screen before closing it.

Immediate depriorities:

Fast Iteration Architecture

Persistent TP1 Workers

Create one long-lived worker per B70. Each worker must:

The first implementation may use a private harness and dispatch table rather than modifying the production server. Full endpoint validation remains a separate promotion step.

Disk And RAM Caches

Build a versioned B70 model-pack format keyed by:

The pack should eventually contain:

Keep the active GGUF and model pack on internal NVMe. Use mmap and the Linux page cache so all four workers share one host copy. Use a bounded pinned staging ring and large asynchronous uploads instead of per-tensor malloc/copy/wait behavior where the backend permits it.

Do not use SYCL_CACHE_PERSISTENT=1 on this B70 system.

Golden Diagnostic State

Persist representative real-model inputs for:

Keep a pristine post-prefill state snapshot in each development worker and reset working state with device-to-device copies. This is valid for diagnostic kernel and cycle iteration only. It is forbidden as evidence for the cold realistic promotion gate.

Kernel Build Cache

Iteration objectives:

Four-GPU TP1 Workflow

Normal development allocation:

Promotion allocation:

Do not interpret four independent TP1 workers as a four-GPU model result.

Validation Ladder

Every candidate advances through these gates:

  1. Saved-tensor parity against golden layer inputs and states.
  2. Real-shape kernel timing, achieved bandwidth/compute, and spill/occupancy inspection.
  3. One-layer replay with exact state transition checks.
  4. Full no-spec target-forward A/B.
  5. Complete MTP or DFlash cycle A/B with acceptance and rollback checks.
  6. Short endpoint canary with fresh state.
  7. Four-GPU paired/swapped realistic-suite crossover.
  8. Exact cases, repeat quality, baseline parity, and longer-context state gate.
  9. Promotion packet, reproduction recipe, and LocalMaxxing preflight if it is a genuine matching record.

Failing a gate closes or redesigns the candidate. A fast microbenchmark does not justify skipping endpoint or state validation.

Phase 0: Preserve And Rebaseline

Deliverables:

Exit gate:

Phase 1: Persistent Runner And Model Pack

Deliverables:

Exit gate:

Phase 2: Full Device Replay

Deliverables:

Exit gate:

Phase 3: Maximum Useful Fusion

Priority fusion families:

  1. output MMVQ + residual epilogue for attention/GDN output and FFN down;
  2. residual + RMSNorm + Q8_1 production for the next projection;
  3. combined gate/up projection with fused SiLU/multiply and down-input quantization;
  4. compatible shared-input QKV projection with RoPE/KV write epilogues;
  5. GDN convolution, gate preparation, state production, and direct persistent state write;
  6. final norm, LM-head reduction, and sampler preparation;
  7. multi-row equivalents for verifier widths.

Preserve the fastest projection producer. Split a fusion that causes material GRF spills, occupancy loss, or less than 1.1x improvement over the unfused boundary.

Exit gate:

Phase 4: True Xe2 Batched Verification

Architecture:

Exit gate:

Phase 5: Fastest Speculation Policy

MTP

Use MTP3 as the reliable baseline. Optimize the complete cycle, not isolated draft kernels:

Stop increasing depth when marginal accepted tokens cost more cycle time than they return.

DFlash

Treat DFlash as the primary high-ceiling lane:

small quantized draft block
  -> one parallel draft forward
  -> one true batched target verification
  -> longest-prefix acceptance and state commit

Use this cycle equation for every configuration:

throughput = accepted output tokens / complete draft+verify+commit time

Do not optimize acceptance without cycle time, or cycle time without accepted tokens. Report both by workload category.

Adaptive Policy

Speculation exit gates:

Phase 6: Reduce Target Bytes

After the Q4 fused speculative pipeline is functioning, evaluate an offline rotated/codebook Q3 model pack. Consider Q2 only as a separately quality-gated research lane.

Exit gates:

Immediate Ordered Checklist

  1. Mark the current protected main-repo paths and dirty llama.cpp MMVQ files as protected work.
  2. Fix the experiment launcher to record current GGML_SYCL_ENABLE_* values.
  3. Add explicit SYCL graph requested/rejected/entered/replayed counters.
  4. Validate and promote the Q4_0 MMVQ rows 1..17 dispatch fix into the experiment ledger.
  5. Run strict no-spec, MTP3, and DFlash baseline rows on current source.
  6. Capture the first low-perturbation target/draft/verify timeline.
  7. Build the golden M=1/4/8/16 activation and state corpus.
  8. Build the persistent one-GPU worker skeleton and native kernel-module ABI.
  9. Implement capture-safe recurrent concat/state assembly and measure full replay.
  10. In parallel, start the real-shape DPAS verification harness and the first output-MMVQ+residual fusion.
  11. Integrate the winning verifier and fusion kernels into a complete MTP cycle.
  12. Re-run DFlash block 5/8/15 economics, then implement adaptive routing.
  13. Promote only through paired/swapped four-GPU strict gates.
  14. Reassess Q3 only after the Q4 cycle is efficient.

Required Durable Artifacts

For each phase, keep:

Large model packs, native modules, and state corpora may stay outside Git under the active benchmark/model storage tree. Track their paths, manifests, checksums, and summaries in the repository.

Definition Of Done

This lane is not done because a microkernel is fast or because a synthetic prompt reaches a high number. It is done when:

This site is open source. Improve this page.