b70-optimization-lab

Gemma 4 Q8 latest llama.cpp c926ad098 A/B

Date: 2026-06-23 Owner/agent: Codex

Hypothesis

Move the current filled-long MTP record identity from llama.cpp dec5ca557 to current upstream c926ad098 to check whether recent server/spec/runtime changes improve Gemma 4 Q8 single-B70 decode. This is a runtime A/B, not a precision change.

Build

Built in a separate worktree so the proven dec5ca557 checkout remains intact:

WT=/home/steve/src/llama.cpp-latest-gemma
BUILD="$WT/build-sycl-b70-aot-bmg-g31"
source /opt/intel/oneapi/setvars.sh --force
cmake -S "$WT" -B "$BUILD" -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=icx \
  -DCMAKE_CXX_COMPILER=icpx \
  -DGGML_SYCL=ON \
  -DGGML_SYCL_F16=ON \
  -DGGML_SYCL_DEVICE_ARCH=bmg-g31
cmake --build "$BUILD" -j 16 --target llama-server llama-cli llama-bench
"$BUILD/bin/llama-server" --version

Build result: version: 9769 (c926ad098).

Shared Run Identity

• model repo: unsloth/gemma-4-26B-A4B-it-GGUF
• filename: gemma-4-26B-A4B-it-UD-Q8_K_XL.gguf
• draft filename: mtp-gemma-4-26B-A4B-it.gguf
• runtime: llama.cpp SYCL AOT BMG build
• backend: SYCL / Level Zero, one full Q8 replica on one B70
• context: 8192
• batch / ubatch: 512 / 64
• KV cache dtype: f16/f16; draft KV f16/f16
• API mode: /v1/chat/completions
• benchmark shape: filled-long, actual 588 prompt tokens / 512 output tokens
• shared flags: --parallel 1 --cache-ram 0 --spec-type draft-mtp --spec-draft-n-max 7 --spec-draft-n-min 2 --spec-draft-p-min 0.12 --no-spec-draft-backend-sampling --spec-draft-threads 32 --spec-draft-threads-batch 32, FLASH_ATTN=off, POLL=50, REASONING=off, GGML_SYCL_DISABLE_OPT=0

Results

Label	Runtime	Extra delta	Gate	tok/s after TTFT	wall tok/s	TTFT ms	Decision
gemma4-q8-gpu0-mtp-n7-latest-c926ad098-nmin2-pmin012-nobs-dthreads32-dtb32-filled-long-deep-20260623T112529Z	c926ad098	default checkpoints	384/384	90.92	68.41	1853.75	Valid, not a record; new upstream checkpoint default hurts TTFT.
gemma4-q8-gpu0-mtp-n7-latest-c926ad098-ctxcp0-nmin2-pmin012-nobs-dthreads32-dtb32-filled-long-deep-20260623T113058Z	c926ad098	--ctx-checkpoints 0	384/384	91.16	71.06	1588.77	New valid high-water mark after TTFT; still worse wall than the old record because TTFT is higher.

Reference old record: gemma4-q8-gpu2-mtp-n7-aot-nmin2-pmin012-nobs-dthreads32-dtb32-filled-long-deep-20260623T101814Z at 91.05 tok/s after TTFT and 82.97 tok/s wall on dec5ca557.

Decision

Promote the checkpoint-off c926ad098 result as the new after-TTFT single-session decode record for this lane because it passes the full chat canary and beats the previous valid high-water mark (91.1567 vs 91.0457 tok/s). Submitted to LocalMaxxing and approved as cmqqkmbhr017oqo017rdfxqh2.

Do not promote the default-checkpoint run. Upstream c926ad098 defaults --ctx-checkpoints to 32; with this benchmark shape, that default inflated TTFT and lowered wall throughput sharply. Future c926ad098 runs should include --ctx-checkpoints 0 unless the experiment is explicitly about checkpointing or long-context reuse.

Follow-Up

The latest runtime did not produce a large speed step. The next high-leverage lane remains source-level MTP draft overhead: the Gemma 4 MTP path copies h_nextn hidden states from target GPU output to host and back into draft batches for each drafted step. That is more promising than more p-min/thread neighbors, which now repeatedly cluster below the record.

This site is open source. Improve this page.