b70-optimization-lab

2026-06-23 15:04Z: MTP draft fast top-k bypass

Goal: reduce llama.cpp draft-MTP overhead without changing the promoted Q8 quality lane. This patch bypasses the generic sampler in the MTP draft loop when backend sampling is disabled and uses a direct logits top-k scan instead.

Baseline identity:

• model: unsloth/gemma-4-26B-A4B-it-GGUF, gemma-4-26B-A4B-it-UD-Q8_K_XL.gguf;
• draft: mtp-gemma-4-26B-A4B-it.gguf;
• runtime: llama.cpp c926ad098, SYCL/Level Zero AOT bmg-g31;
• shape: BENCH_PROMPT_MODE=filled-long, actual 588 prompt / 512 output;
• flags: MTP_N_MAX=7, MTP_N_MIN=2, MTP_P_MIN=0.12, MTP_BACKEND_SAMPLING=0, MTP_DRAFT_THREADS=32, MTP_DRAFT_THREADS_BATCH=32, MTP_EXTRA_ARGS='--ctx-checkpoints 0';
• prior official best: 91.156717 tok/s after TTFT, LocalMaxxing cmqqkmbhr017oqo017rdfxqh2.

Patch artifact:

• patches/gemma4-llamacpp-mtp-draft-fast-topk-20260623.patch

New source knobs:

• LLAMA_MTP_DRAFT_FAST_TOPK=1: enable direct-logit MTP draft top-k path;
• LLAMA_MTP_DRAFT_TOP_K=<n>: controls the local top-k candidate set;
• existing LLAMA_MTP_DRAFT_LOGIT_GAP_MIN remains available but was not used in the promoted result.

The fast path is only used when backend sampling is disabled. It greedily drafts the top-1 token from the direct logits scan, computes the existing p_min gate from a top-k-local softmax, and leaves the verifier/canary discipline unchanged.

Initial 4-GPU Sweep

Stamp: 20260623T150436Z

GPU	Label suffix	Fast top-k	Canary	tok/s after TTFT	Wall tok/s	TTFT ms	Delta vs prior
0	fasttopk2	2	384/384	91.138379	71.005378	1592.883	-0.018339
1	fasttopk4	4	384/384	90.385943	70.434578	1604.691	-0.770775
2	fasttopk10	10	384/384	91.276506	71.049234	1596.988	+0.119788
3	fasttopk20	20	384/384	90.987971	70.883177	1596.090	-0.168746

Interpretation: top_k=10 produced a tiny first-pass win, but the gain was too small for submission without a repeat. top_k=2/4/20 were losses.

Exact top_k=10 Repeat

Stamp: 20260623T150833Z

GPU	Canary	tok/s after TTFT	Wall tok/s	TTFT ms	Delta vs prior
0	384/384	91.618942	71.287464	1593.881	+0.462225
1	384/384	90.479268	70.541099	1599.384	-0.677450
2	384/384	90.861494	70.728135	1603.829	-0.295224
3	384/384	91.226231	71.012737	1597.626	+0.069514

Decision: promote GPU0 repeat as the new single-B70 Q8 filled-long record. The repeat cleared the previous record by about 0.46 tok/s and passed the promotion-depth canary.

Promoted run:

• label: gemma4-q8-gpu0-mtp-n7-c926-fasttopk10-repeat-ctxcp0-nmin2-pmin012-nobs-dthreads32-dtb32-filled-long-deep-20260623T150833Z;
• summary: data/gemma4-q8-gpu0-mtp-n7-c926-fasttopk10-repeat-ctxcp0-nmin2-pmin012-nobs-dthreads32-dtb32-filled-long-deep-20260623T150833Z/summary.json;
• benchmark: data/gemma4-q8-gpu0-mtp-n7-c926-fasttopk10-repeat-ctxcp0-nmin2-pmin012-nobs-dthreads32-dtb32-filled-long-deep-20260623T150833Z/p512o512.json;
• canary: data/gemma4-q8-gpu0-mtp-n7-c926-fasttopk10-repeat-ctxcp0-nmin2-pmin012-nobs-dthreads32-dtb32-filled-long-deep-20260623T150833Z/chat-canary.json;
• LocalMaxxing: cmqqsecuk01azqo018ahv0i1s;
• queue: data/localmaxxing-gemma4-26b-a4b-q8-b70-mtp-n7-c926ad098-fasttopk10-filledlong512-20260623.queue.json;
• response log: data/localmaxxing-responses/gemma4-26b-a4b-q8-b70-mtp-n7-c926ad098-fasttopk10-filledlong512-20260623.submit.log.

Follow-Up

Do not retest plain top_k=2/4/20 in this exact identity unless another patch changes the draft-loop cost model. The useful neighborhood is now:

• LLAMA_MTP_DRAFT_FAST_TOPK=1, MTP_DRAFT_TOP_K=10 plus MTP_P_MIN neighborhood (0.115, 0.120, 0.125, 0.130);
• top_k=8/10/12/16 repeat with the same fast path if the p-min neighborhood does not move;
• source-level draft-loop work that reduces draft llama_decode overhead, because sampler bypass produced only a modest win.

This site is open source. Improve this page.