b70-optimization-lab

2026-06-24T1045 QAT GGUF Side Lane

Reason

The primary Q8/INT8-quality lane appears plateaued:

Small Q8 MTP knobs, draft quant variants, Q8_0 target, Vulkan, vLLM INT8 smoke, direct argmax IDs, device hidden-state handoff, and direct unroll were explored in 20260624T0235-mtp-fused-unroll-feasibility.md; direct-unroll7 plus q-only assistant inputs advanced the Q8 fresh record from 95.263 to 96.822 tok/s, and a later BATCH_SIZE=1024 / UBATCH_SIZE=1024 / THREADS=8 tune advanced it again to 98.491 tok/s; this remains far below the >150 tok/s research target.

Later update: the Q8 target/verifier lane subsequently advanced to 103.2992004295621 tok/s fresh row0 with selected-softmax/weighted-sum MoE guards (cmqsylo2l011nqr011yydjvne). Keep the QAT/Q4XL results in this note as lower-precision side-lane evidence, not as the Q8 headline.

This side lane tests whether the QAT-trained 4-bit family can deliver a major fresh-response speed jump while preserving acceptable quality. It is not the default Q8/INT8 headline lane.

Fresh-Response Validity Rule

Only row 0 of a no-cache benchmark can be used as a headline unless every row is a distinct fresh prompt with no usable prior generated continuation. Repeated prompt warmed/history throughput, n-gram continuation learning, prefix/cache reuse, response reuse, and averaged repeated-output speedups are not valid fresh-response headline numbers.

For this lane, every candidate must record:

Downloaded Candidate

Repository:

unsloth/gemma-4-26B-A4B-it-qat-GGUF

Files being downloaded to:

/mnt/fast-ai/llm-models/gemma4-26b-a4b-it-qat-gguf/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
/mnt/fast-ai/llm-models/gemma4-26b-a4b-it-qat-gguf/MTP/gemma-4-26B-A4B-it-Q4_0-MTP.gguf

Checksums after download:

dcf179a91153e3a7ece792e48ef872180d9d6ef9b7677f0a0bd3e83cfe624d5e  gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
62bd3af7f66c9308de9a5454233852f8c7324c93767e8dfb824ed45b9179864a  MTP/gemma-4-26B-A4B-it-Q4_0-MTP.gguf

Planned first smoke, once the files are complete:

Promotion rule: even if this exceeds the Q8 record, treat it as QAT/Q4 side-lane throughput until a separate quality decision says it is acceptable to publish as a quality-equivalent result. Do not submit it to LocalMaxxing as the Q8/INT8 lane.

Operational Note: One GPU Per Replica

For this QAT lane, each run is pinned with ONEAPI_DEVICE_SELECTOR=level_zero:N. After that selector is applied, llama.cpp sees the selected physical GPU as SYCL0. Therefore replica runs on physical GPU 1/2/3 must still use:

LLAMA_DEVICES=SYCL0
--spec-draft-device SYCL0

Using SYCL1, SYCL2, or SYCL3 inside the filtered process produced invalid device failures.

First QAT Screen

All results below are fresh-response row-0 numbers unless noted. cached_tokens was 0 for promoted comparisons. Output SHA matched the current deterministic reference (d3236ebed08dda8f19a0fec78622967b3622704da06819c98a7e0e63f90d982b) for passing canary runs.

Baseline target-only:

MTP first screen:

Acceptance detail from server logs:

QAT Depth Sweep

CANARY_REPEATS=64, one fresh p512/o512 row:

Conclusion: n7 remains the best depth. More depth improves nominal acceptance on some requests, but the extra draft work dominates.

Runtime / Memory Sweep Around QAT n7

Direct-Unroll Screen Before Final-h_nextn Patch

This used LLAMA_MTP_DRAFT_DIRECT_ARGMAX_IDS=1 and LLAMA_MTP_DRAFT_DIRECT_ARGMAX_UNROLL=N on the QAT n7 config.

Conclusion: direct unroll had the right shape but stayed below scalar QAT n7. The next source patch under validation skips the final unused h_nextn projection/export inside the direct-unroll graph. This is intended to reduce one nextn_proj_post matmul on the terminal unroll step without changing sampled token outputs.

Direct-Unroll Final-h_nextn Patch Result

Patch artifact:

patches/gemma4-26b-a4b-q8-b70/20260624T1120-llamacpp-gemma4-assistant-directunroll-skip-final-hnext.patch
sha256: 184323a276a892d2750124bcbd92b5ba6d982e13d530bacb67337b193538bd5f

The patch avoids computing/exporting h_nextn for the terminal direct-unroll step in src/models/gemma4-assistant.cpp. It is semantically scoped to LLAMA_MTP_DRAFT_DIRECT_ARGMAX_UNROLL > 1.

Post-patch results:

Conclusion: valid loss. The patch is directionally positive but too small and direct-unroll remains below scalar QAT n7 (131.58998767789097). Preserve the patch artifact for reference, but do not promote it as the active optimization.

Argmax / Verifier Screen

After direct-unroll lost, a scalar QAT n7 control and three low-risk argmax/verify variants were tested. CANARY_REPEATS=64, one fresh p512/o512 row.

Conclusion: no argmax/verify shortcut beats scalar QAT n7. The scalar control reproduced the current best at 131.7 tok/s, but the >150 tok/s target needs a real reduction in draft/verify work, not another sampler knob.

Direct-Unroll Graph-Reserve Patch

A second patch increased graph_max_nodes() only for Gemma4 assistant MTP direct argmax unroll (ctx_type == MTP, direct argmax IDs enabled, unroll > 1). This tested the hypothesis that unroll7 was failing because scheduler graph reserve was undersized, not because the optimization was semantically invalid.

Result: the patch fixed the crash, but did not improve throughput enough.

Conclusion: graph-reserve patch is useful as a bug fix for testing unroll7, but direct unroll still loses to scalar QAT n7. Do not promote it as a performance win. The next useful step is phase profiling around scalar QAT n7 to identify where the ~132 -> >150 gap remains.

Scalar QAT n7 Profile And Device-H Screen

Profile run:

data/gemma4-qatq4xl-gpu0-mtp-n7-scalar-profile-20260624T113430Z/
server log: /mnt/fast-ai/bench-results/gemma4-26b-a4b-q8/servers/gemma4-qatq4xl-gpu0-mtp-n7-scalar-profile-20260624T113430Z.server.log

Canary passed (32 rows in the short profile run) and fresh throughput was 133.1792477533177, so profiling did not materially distort the result.

Profile totals at the end of the run:

mtp decode phase profile:
  calls=904, tokens=905, total_ms=864.874, per_call_ms=0.957
  process_ubatch_ms=418.935
  post_extract_ms=430.189
  balloc_ms=2.367, sched_reserve_ms=0.713, init_batch_ms=11.280

draft-mtp:
  draft_decode_ms=854.781
  fast_scan_ms=79.482
  hidden_get_ms=0.459
  handoff_ms=0.755
  draft_decodes=903

Interpretation: the scalar QAT n7 draft path is split roughly in half between assistant graph compute/submit (process_ubatch) and output extraction (post_extract). Sampler/top-k and hidden-row bookkeeping are already small. The remaining >150 tok/s gap likely needs lower-level output extraction or draft graph work reduction, not another sampler knob.

After adding finer extraction counters, scalar QAT n7 showed:

data/gemma4-qatq4xl-gpu0-mtp-n7-scalar-profile-split-20260624T114353Z/
fresh: 134.46824800804353
post_extract_ms=344.677
logits_extract_ms=338.839
h_nextn_extract_ms=5.695
sampled_extract_ms=0.021

So full raw-logit extraction is the dominant post-extract cost. Direct IDs were profiled as a comparison:

data/gemma4-qatq4xl-gpu0-mtp-n7-directids-profile-split-20260624T114443Z/
fresh: 131.3893864046233
post_extract_ms=516.040
logits_extract_ms=0.009
h_nextn_extract_ms=512.904
sampled_extract_ms=3.007

This means the direct-ID graph path removes host logits transfer, but the graph work / first remaining extraction waits longer overall. Do not pursue direct IDs further until there is a better on-device argmax/top1 kernel.

Device hidden-state handoff alone was then tested without direct IDs:

This is a small valid win over prior scalar controls, but still far below >150 tok/s. It suggests hidden-state host extraction is not the main blocker; the expensive part is broader output extraction / decode envelope.

Split profile with device-H:

data/gemma4-qatq4xl-gpu0-mtp-n7-deviceh-profile-split-20260624T114555Z/
fresh: 133.70532508555132
post_extract_ms=334.508
logits_extract_ms=334.344
h_nextn_extract_ms=0.020

Device-H removes host h_nextn extraction, but logits extraction remains the dominant cost.

Device-H depth sweep:

Conclusion: device-H does not make deeper MTP viable. n7 remains the only useful depth.

Draft quantization screen with QAT target + device-H:

Conclusion: smaller Q2/Q3 draft files do not produce faster draft kernels here; acceptance/cost balance is much worse. Keep the QAT Q4_0-MTP draft.

Draft-only KV cache type screen with QAT target + device-H:

Conclusion: draft KV quantization is not useful here. Without FA it collapses; with FA it remains below f16 draft KV. Keep draft KV at f16.

Fast top-k confidence screen:

Earlier p_min sweeps with LLAMA_MTP_DRAFT_FAST_ARGMAX=1 were not real confidence-stopping tests because the fast-argmax path assigns p=1.0, so p_min never stops drafting. A real fast-top-k screen was run with device-H:

Conclusion: top-k confidence stopping is slower than fast argmax on this stack. The extra top-k/probability work costs more than it saves by stopping marginal draft positions.

Poll sweep on current scalar/device-H config:

Conclusion: flat/no win. Keep POLL=100 or 75; neither changes the plateau.

Thread sweep on current scalar/device-H config:

Conclusion: flat/no win. Keep the prior THREADS=16, --spec-draft-threads 32, --spec-draft-threads-batch 32 recipe.

F16 draft-logit extraction patch screen:

Conclusion: failed optimization. The patch reduced the draft-logit transfer width, but the host-side fp16 conversion / scan path costs more than it saves. It is canary-clean but slower than the scalar/device-H baseline, so keep it as a negative artifact and do not promote it.

Patch artifact:

Context / batch shape sweep on current scalar/device-H config:

Conclusion: no decode-throughput win. Smaller ctx can reduce TTFT in one shape (ctx=2048, batch=1024, ubatch=1024 got ttft=0.665s), but the fresh decode plateau remains ~131-132 tok/s. Keep ctx=8192, batch=512, ubatch=512 for comparability unless optimizing TTFT separately.

Source-audit follow-up: redundant syncs in scalar fast argmax.

Explorer audit found the current scalar fast-argmax path was doing more synchronization than intended:

  1. draft_fast_argmax_sample() explicitly calls llama_synchronize(ctx_dft) after llama_decode();
  2. it then probed llama_get_logits_f16_ith(), whose public wrapper also synchronizes, even when f16 draft-logit extraction is disabled;
  3. it then called llama_get_logits_ith(), whose public wrapper synchronizes again.

Patch under test:

Correctness expectation: unchanged draft tokens and verifier behavior; this only removes redundant synchronization / dead f16 probing after the decode has already been synchronized.

Four-way screen launched:

Result:

Conclusion: no-sync is canary-clean but not a win; non-QAT repo MTP drafts are significantly slower when paired with the QAT target. The redundant public wrapper syncs were not the bottleneck at the current plateau; draft logits extraction and host scan remain the dominant cost.

Patch artifact:

Follow-up screen after rebuilding with the SYCL_ARGMAX_BLOCK_SIZE=512 change and no-sync scalar accessors:

Conclusion from this follow-up: the current plateau is not caused by redundant public API syncs, Level Zero copy-engine policy, or the SYCL argmax block size. The valid fresh-response QAT/Q4 path remains the scalar fast-argmax + device-H-handoff lane at about 133 tok/s. The unstable direct-ID/unroll routes should stay as negative artifacts unless a deeper sampled-token buffer ownership/indexing bug is fixed.

Small scalar/device-H parameter sweep, keeping the stable path and changing only draft minimum length or draft thread counts:

Conclusion: no scalar/runtime parameter win. n_min=2, draft threads 32/32, POLL=100, ctx=8192, batch=512, ubatch=512, FA off remains the best validated scalar/device-H shape for this side lane.

Separate fresh draft-model screen with downloaded E4B Q4_0 GGUF:

Conclusion: true separate-draft speculation is valid fresh-response work, but not useful here. llama.cpp draft-simple repeatedly decodes the full E4B draft model and samples through the generic draft-simple path, so it is far slower than the tiny Gemma4 MTP assistant. Do not pursue E4B/12B draft-simple unless the draft-simple implementation itself is redesigned.

SYCL argmax512 consistency fix:

Post-fix retest:

Conclusion: the argmax fix is a real correctness cleanup (direct-ID/unroll no longer crash), but not a throughput win. Backend argmax remains slightly slower than exporting logits and doing the scalar host scan on this workload.

Post-fix direct-unroll depth sweep:

Conclusion: no direct-unroll win. The deeper direct-unroll graph must amortize more MTP steps before it approaches scalar, but even unroll7 post-fix only reached 132.36125747200137, still below the scalar/device-H best.

SYCL argmax1024 retest:

After the argmax kernel consistency fix, SYCL_ARGMAX_BLOCK_SIZE was raised from 512 to 1024 and the four most relevant lanes were rebuilt/retested. This is a valid test of the larger-block argmax kernel, not a Q8 headline result. CANARY_REPEATS=64; all throughput values below are fresh row-0 p512/o512 with cached_tokens=0.

Conclusion: argmax1024 is canary-clean and produces the best measured QAT/Q4 side-lane row so far (133.27420695985995), but the delta over the previous best (133.13947504649965) is only +0.1347 tok/s and should be treated as noise-level/marginal unless repeated. It does not change the main conclusion: the QAT/Q4 lane remains below the >150 tok/s target, and it is not the primary Q8/INT8-quality headline lane.

Patch artifact:

Gemma4Assistant q-only attention input patch:

Patch idea: Gemma4Assistant uses shared target K/V and calls build_attn(..., k_cur=nullptr, v_cur=nullptr, ...), but the generic ISWA input builder still creates and uploads K/V index tensors. An env-gated patch (LLAMA_GEMMA4_MTP_QONLY_ATTN_INPUTS=1) nulls those unused K/V index inputs for the assistant graph while preserving masks and RoPE rotation inputs.

Patch artifact:

Q8-quality screen:

Early conclusion for Q8: q-only alone was not a win in this scalar screen. Later, q-only combined with direct-unroll7 did become the current promoted Q8 record (98.491 tok/s, cmqs56wv100kjqr01de3fdspd). Keep this distinction: the patch is useful as part of the direct-unroll stack, not as a standalone scalar-path improvement.

QAT/Q4 screen:

Follow-up around direct-unroll7 q-only:

Conclusion: q-only attention inputs are canary-clean and give the best measured QAT/Q4 side-lane row so far (134.37693544373184), but the identical repeat fell to 133.25860672997848, so treat this as a marginal/noisy side-lane improvement, not a stable breakthrough. It does not solve the >150 tok/s fresh-response target and does not help the Q8-quality lane.

Single-server confirmation:

The four-way sweeps above run four llama-server processes concurrently and can understate single-session throughput because each server requests substantial CPU-side runtime support (THREADS=16, HTTP workers, and SYCL host work) on the same 16-core host. A single-server confirmation of the best QAT/Q4 shape:

Conclusion update: use the single-server no-profile row (136.05802749417853) as the best solid QAT/Q4 side-lane result. Use four-way sweeps for fast research screening, but confirm any candidate alone before making a headline throughput claim.

Runtime screen around q-only direct-unroll7:

Conclusion: no runtime knob win. The remaining gap is inside assistant graph execution (process_ubatch_ms), not flash attention, VMM, Level Zero copy policy, or CPU thread count.

Current Best Valid Result In This Side Lane

Best QAT/Q4 side-lane result:

data/gemma4-qatq4xl-gpu0-mtp-n7-directunroll7-qonly-single-20260624T1410Z/
fresh tok_s_after_ttft: 136.05802749417853
canary: pass, 256 rows completed
cached_tokens: 0
target: QAT UD-Q4_K_XL
draft: Q4_0-MTP

This is a valid fresh-response result for the QAT/Q4 side lane, but it is not a Q8/INT8-quality headline and should not be mixed with the primary Q8 record. It also does not meet the user target of >150 tok/s.

This site is open source. Improve this page.