b70-optimization-lab

Reproducibility Map

This is a durable catalog of runnable recipes and promoted evidence. It is not the authority for the currently loaded service or active research lane; use CURRENT.md for that live state. Historical service claims below describe recorded lane context and may not describe what is running now.

This page connects the active Gemma 4 service, the deployable MiniMax baseline, the session-cache experiments, the TurboQuant patch, and the long-context research path. It is meant for a fresh human or agent who needs to reproduce or review the current work without reading every historical note first.

Hardware scope: the local Intel lab is four Arc Pro B70 32 GB GPUs (128 GB aggregate VRAM). Results here are useful because they are produced on real community-accessible XPU hardware, but the same limit also constrains larger model coverage. Additional high-VRAM Intel hardware would let this map include larger GLM/DeepSeek-class lanes and more concurrent service/optimization comparisons without sacrificing the current endpoint. The lab has spare EPYC 9015 PCIe 5.0 slot capacity, so the missing piece for broader Intel coverage is higher-memory XPU hardware rather than host expansion.

What Is Production Today

As of 2026-07-08, the active LAN endpoint on this host is temporarily the Gemma 4 26B Q8 coding-agent service:

Restore or stop it from docs/gemma4-26b-q8-service-runbook.md.

The usual model-slot production profile to restore after this temporary service is the Gemma 4 c8 profile:

Restore production Gemma 4 c8:

cd /home/steve/llm-optimizations
printf '%s\n' "/'" | sudo -S -p '' \
  scripts/switch-vllm-model-slot.sh switch gemma4-12b-it-int4-autoround-c8

Current Gemma 4 recipe and results:

Latest full-32K concurrency conclusion:

Active Optimization Lane

Qwen3.6 27B INT4 AutoRound is the current optimization target, separate from the production LAN endpoint:

Laguna S 2.1 INT4 Qualified Published Result

The four-B70 Laguna row is approved at 102.971435596 tok/s under the submitted legacy 100-event/99-interval convention. The conventional rate from the same timestamps is 101.941721240 tok/s, so the conventionally counted 102 objective remains short by 0.058278760 tok/s. The fixed cold suite used exact width 12, DFlash depth 11, and the audited 146/145 Breakable PIECEWISE topology. The first valid preregistered score was 13/13 token-and-text exact against the canonical q1 teacher, cache-zero on every row, and approved by LocalMaxxing as cms2ccv2d00lps201rej94pjy.

Use:

The record uses BF16 KV to preserve its BF16 canonical-teacher contract. Poolside’s quantized checkpoint officially specifies calibrated FP8 KV, which is now an active, separately labeled experiment under experiments/laguna-s-2.1-fp8-kv-xpu-b70/. It has its own source patches, checkpoint/runtime scale audit, FP8 q1 teacher, and promotion gates; it is not a silent record-lane substitution.

The repro tracks the sealed raw benchmark, log, environment, cleanup, and idle evidence; portable release-only model hashes; the complete observed runtime; and every direct or transitively loaded native library. Exact lab replay and a source-equivalent clean rebuild are separate claims: a rebuild with different binary hashes is a new environment and must pass all semantic, topology, teardown, and performance gates.

DeepSeek V4 Flash K160 Closed Frontier

The four-B70 DeepSeek V4 Flash experimental uniform-K160 lane is paused. Its best verified one-active-generation result is 80.820052 tok/s with target-verified DSpark7; three independent strict medians were 80.820052, 76.900178, and 78.287226 tok/s. All 36 realistic rows were cache-zero and 24/24 ordered exact canaries passed. LocalMaxxing approved cmrquta9905w3lg013m5vxoqx.

Use the closed result packet and standalone pinned repro. The recipe includes exact source bundles because the measured vLLM, XPU kernel, and oneCCL commits are local experimental history rather than upstream commits. The K160 artifact is hash-pruned and its calibration/ranking is not reproducible, so this record applies only to that explicitly labeled artifact.

MiniMax Deployable Baseline

The MiniMax 32K FP16-family KV c1 endpoint remains the deployable baseline recipe and optimization reference:

Fresh install guide:

../repro/minimax-m27-b70-110tps-ubuntu24-20260523/README.md

Human deployment guide:

b70-minimax-ubuntu24-deployment.md

Main server script:

../repro/minimax-m27-b70-110tps-ubuntu24-20260523/scripts/06-serve-openai-compatible.sh

Operational profile switcher:

../experiments/minimax_xpu_kv_offload/scripts/switch_session_cache_profile.sh

Restore c1:

cd /home/steve/llm-optimizations
experiments/minimax_xpu_kv_offload/scripts/switch_session_cache_profile.sh c1

Check status:

experiments/minimax_xpu_kv_offload/scripts/session_cache_status.sh

Gemma 4 26B Realistic-Suite Observation

The current Gemma 4 26B A4B Q8 single-B70 realistic-suite observation is documented in the result packet:

../results/gemma4-26b-a4b-q8-b70/reproduce.md

Current handoff and production backend recipe:

This is a localhost llama.cpp backend recipe, not the current public :8000 systemd frontdoor profile. The tracked systemd unit ../deploy/systemd/gemma4-26b-q8-llamacpp.service starts the backend on 127.0.0.1:19350; wire the frontdoor to that backend only after smoke passes.

Standalone current repro:

../repro/gemma4-26b-a4b-q8-b70-125tps-20260701/README.md

Use it when the goal is to reproduce the current fixed-suite cold-response frontier rather than the older synthetic filled-long diagnostics. The older standalone ../repro/gemma4-26b-a4b-q8-b70-95tps-20260624/README.md folder remains as a superseded 95.264 tok/s reproduction artifact.

Record identity:

The full optimization ledger remains in ../results/gemma4-26b-a4b-q8-b70/README.md.

Baseline Build Inputs

The fresh Ubuntu 24 repro builds from source and applies two compressed patch artifacts from the older strict-speed repro:

The build script decodes and applies those patches automatically:

../repro/minimax-m27-b70-110tps-ubuntu24-20260523/scripts/03-build-stack.sh

Pinned source commits are listed in:

../repro/minimax-m27-b70-110tps-ubuntu24-20260523/README.md

Live-source audit snapshots from the originating machine are also tracked:

These snapshots capture the dirty local /home/steve/src/vllm and /home/steve/src/llm-scaler trees after the session-cache and TurboQuant research. Treat them as review/audit artifacts, not as clean upstream-ready patches. The clean fresh-install repro still uses the two compressed promoted patch bundles listed above.

Baseline Results

The fresh deployable baseline records:

Tracked summaries:

Detailed notes:

Session-Cache / RAM-Backed Juggling

This is the main experimental path for keeping multiple long conversations warm. It is not one huge active context.

Mental model:

Entry points:

Scripts:

Current operational recommendation:

Near-full c2 validation:

Known-good c2 operations smoke:

The operations smoke is intentionally smaller and cleaner. It does not define the desired c2 context ceiling; c2 should be presented as a 32K-window profile.

Result file from the originating host:

/mnt/fast-ai/bench-results/minimax-m27-b70-serve/session-cache-c2-ops-fact-900lines-20260525T223527Z.json

The raw /mnt/fast-ai file is not in GitHub; the result is summarized in:

../experiments/minimax_xpu_kv_offload/notes-20260525-session-cache-operations.md

Concurrency/sustained decode notes:

Headline sustained warm results:

Live c4 caveat:

TurboQuant

TurboQuant is a compressed-KV research lane. It can raise the live KV ceiling, but it is not the production mode.

Patch artifact:

../patches/vllm-turboquant-xpu-workspace-fallback-20260525.patch

Repro script:

../scripts/repro-minimax-turboquant-xpu-workspace-bug.sh

Detailed notes:

Current status:

Important boundary:

TurboQuant plus CPU KV offload still requires the active request’s working KV blocks to fit in live GPU memory. It helps capacity, but it is not active-context overflow.

Full 196K Active Context Path

The credible exact-quality path is CPU-paged attention, not simply increasing --kv-offloading-size.

Design notes and probes:

Experimental patches:

Current design direction:

  1. Keep recent/current KV in normal GPU KV blocks.
  2. Keep older logical KV blocks in CPU offload storage.
  3. Stage old CPU-resident KV chunks into GPU scratch.
  4. Run attention over each chunk.
  5. Merge partial attention outputs using log-sum-exp/LSE state.
  6. Merge old-context attention with normal attention over the live GPU suffix.

This is still a research path, not a serving recipe.

What GitHub Does Not Include

GitHub has:

GitHub does not include:

When a note references a raw /mnt/fast-ai log or JSON, use the summarized values in GitHub unless you are on the originating machine.

This site is open source. Improve this page.