b70-optimization-lab

Model Optimization Guide For AI Agents

This guide is for an AI agent starting a new model optimization lane in this repo. It describes the full path from first contact with a model to a promoted, reproducible result, including what to try, what to avoid, where to look, and how to keep the work useful for future agents.

Use this guide with:

For a concise, evidence-linked catalog of transferable wins and important boundaries, see Cross-Model Patterns Worth Reusing.

0. Operating Principles

Do not start by changing flags or code. Start by making the lane measurable.

The strongest optimization work in this repo came from this loop:

  1. Define the quality lane and benchmark identity.
  2. Establish a valid baseline.
  3. Create a repeatable harness.
  4. Use cheap screens to reject bad ideas.
  5. Use realistic cold-suite gates before promotion.
  6. Record every meaningful result, including failures.
  7. Promote only clean, reproducible, quality-preserving wins.

The weakest work came from:

1. Start With The Workspace

Use the active checkout:

cd /home/steve/llm-optimizations
git status --short --branch
git log -1 --oneline

Expected policy:

Before running or editing anything, read:

AGENTS.md
AGENT_HANDOFF.md
CURRENT.md
docs/model-effort-index.md
docs/research-workflow-playbook.md
docs/localmaxxing.md
docs/local-ops.md

If the repo is dirty:

  1. Identify tracked changes, staged changes, and untracked files.
  2. Decide whether each untracked file is valuable evidence, transient output, a secret, a model artifact, or a build artifact.
  3. Track valuable compact artifacts.
  4. Add ignore rules for transient output, secrets, weights, and logs that are too large or not useful.
  5. Do not delete another agent’s work unless it is clearly transient and already captured elsewhere.

2. Create A Model Lane Packet

Every serious model effort should have a packet under:

results/<model>-<quant>-<hardware>/

Start with these files:

results/<model>-<quant>-<hardware>/README.md
results/<model>-<quant>-<hardware>/reproduce.md
results/<model>-<quant>-<hardware>/validity-gates.md
results/<model>-<quant>-<hardware>/runtime-plan.md
results/<model>-<quant>-<hardware>/model-options.md
results/<model>-<quant>-<hardware>/bugs-failed-paths.md
results/<model>-<quant>-<hardware>/localmaxxing-and-targets.md

Use experiments/<model>-.../ for active research lanes. Use repro/<model>-.../ only when a result is promoted enough to deserve a standalone runnable recipe.

The packet should answer these questions:

3. Lock The Quality Lane

Decide the quality promise before chasing speed.

Examples:

Record:

Do not silently switch quality lanes. The Gemma 26B work was almost derailed by confusion between UD-Q8_K_XL and Q8_0. If the target quality changes, create a labeled side lane.

4. Define Validity Before Benchmarking

Each model needs validity gates. At minimum, define:

For fresh-response headline throughput, use this rule:

Synthetic prompts are allowed for diagnostics. They are not headline throughput unless they pass the same fresh-response policy and are representative of the claimed workload.

5. Establish The Baseline

Before optimizing, run the simplest credible baseline.

For a llama.cpp lane:

For a vLLM lane:

Baseline notes should include:

If the baseline is too slow, do not skip validity. A slow valid baseline is the anchor that prevents false wins.

6. Build The Harness Before The Optimization

A good harness should:

Use consistent folders:

scripts/      shared reusable tools
repro/        promoted runnable recipes
data/         compact structured run outputs
notes/        chronological notes
patches/      source/config diffs
experiments/  active non-promoted research
results/      promoted or closed model packets

Every run label should include model, relevant mode, GPU, and timestamp.

Good label examples:

gemma4-q8-gpu0-125repro-20260702T231635Z
qwen36-tp4-piecewise-forcedcomm-deepgate-20260615
minimax-m27-tp4-moefullforward-strict-20260519

7. Run Four GPUs As A Research Multiplier

On the current host, prefer one full replica per B70 when the model fits.

Use four GPUs for:

Do not use four-GPU screens alone to promote near-record wins. They are good for rejecting bad ideas and finding promising candidates. Promotion should use a clean solo run or a balanced same-window A/B that can survive variance.

Example lane layout:

GPU0: control A
GPU1: candidate A
GPU2: candidate B
GPU3: control B

Then swap candidates and controls in the next wave if the result is close.

8. Measure Variance Early

Before chasing sub-1% wins, measure same-recipe variance.

Do:

Gemma 26B Q8 lesson:

For target-side changes that cannot affect speculation:

  1. Run lower-variance no-spec A/B.
  2. If the no-spec A/B shows a real positive, rerun the normal realistic MTP gate.
  3. Promote only if the normal gate passes and the total result improves.

9. What To Try First

The best early experiments are cheap, reversible, and easy to explain.

Runtime And Launcher Identity

Try:

Avoid:

Quantization And Model Variants

Try:

Avoid:

Single GPU Before Tensor Parallelism

If the model fits on one B70:

TP can help capacity, but PCIe collectives can erase decode gains at batch 1.

Prompt Processing And Context

Try:

Keep prompt-processing wins separate from short-decode headline records unless the same config passes both.

KV-Cache Precision Is A Separate Lane

Do not infer KV precision from weight quantization or torch_dtype. Inspect the checkpoint’s KV-specific scheme, the resolved engine config, and the attention backend actually selected.

For every BF16/FP16 versus FP8/Q8/TurboQuant comparison:

FP8 halves the K/V payload in principle, but that does not guarantee a speedup. At short context, quantize/dequantize work or a weaker backend can dominate. Laguna’s direct B70 screen doubled capacity but was 4.132% slower than BF16 on its early matched-DFlash short-context lane. See the Laguna KV-cache decision.

Speculation

Try:

Avoid:

Kernel And Source Work

Look for:

Use profiling where possible before writing patches. If profiling is hard, add low-risk instrumentation behind an environment flag and remove or default it off after the experiment.

10. What To Look For In Logs And Code

For vLLM/XPU:

For llama.cpp/SYCL:

In source, search first for:

rg -n "spec|draft|argmax|sample|lm_head|MUL_MAT|Moe|MoE|expert|fattn|Flash|graph|cache|restore|sync"

Use rg --files and rg before slower search tools.

11. Diagnose Correctness Failures

Classify failures by signature:

Then map signature to likely causes:

Do not fix before identifying state ownership. For model-state bugs, list every buffer read by decode and every buffer saved/restored/advanced by speculation.

12. Promotion Checklist

Before calling a result promoted:

If any item fails, label the result as diagnostic, invalid, support-only, or inconclusive.

13. LocalMaxxing Submission Rules

Submit only policy-compliant records.

Do not submit:

When submitting, include:

Credential handling is in docs/localmaxxing.md. Never print or commit the API key.

14. Documentation And Artifact Hygiene

Every meaningful experiment should leave behind:

Use compact summaries for Git. Large logs can stay outside Git if a summary records their path and the reason they matter.

Recommended artifact layout:

data/<label>/summary.json
data/<label>/<suite>.json
experiments/<model>/sweeps/YYYYMMDD-<idea>.md
patches/<model>/<YYYYMMDD>-<idea>.patch
results/<model>/README.md
results/<model>/reproduce.md
results/<model>/bugs-failed-paths.md

When a result is promoted, create or update:

repro/<model>-<hardware>-<headline>-<date>/README.md
repro/<model>-<hardware>-<headline>-<date>/run.sh

Keep failed patches unless they were superseded by a clearly linked fix. Failed work prevents future agents from rediscovering the same dead end.

15. Suggested Start-To-Finish Workflow

Use this as the default operating sequence for a new model.

Phase A: Orientation

  1. Read current workspace docs.
  2. Check git state.
  3. Create a model lane packet.
  4. Identify quality lane and model variants.
  5. Identify best public/local baseline.
  6. Decide runtime candidates: llama.cpp, vLLM, OpenVINO, Ollama, or other.
  7. Write validity gates before benchmarking.

Phase B: Baseline

  1. Download or verify model files.
  2. Build or select the runtime.
  3. Launch the simplest server.
  4. Run canaries.
  5. Run fixed realistic suite.
  6. Save summary and logs.
  7. Document baseline and blockers.

Phase C: Harness

  1. Add a wrapper script for the lane.
  2. Record all launcher identity fields.
  3. Add GPU/port/label overrides.
  4. Add canary and benchmark calls.
  5. Ensure clean shutdown.
  6. Add JSON summary output.
  1. Run four independent screens across GPUs.
  2. Test runtime flags, batch/ubatch, graph, FA, VMM, context limits.
  3. Reject obvious losses quickly.
  4. Record negative results.
  5. Keep quality canaries on, even in screens when cheap enough.

Phase E: Profiling

  1. Profile the current best.
  2. Identify top hotspots.
  3. Pick source changes tied to hotspots.
  4. Add default-off flags for risky patches.
  5. Build and run focused A/B.

Phase F: Variance Control

  1. Measure same-recipe variance.
  2. Use no-spec calibration for target-only patches.
  3. Use same-window paired A/B for small deltas.
  4. Capture temperature/frequency if variance is unexplained.
  5. Refuse to promote noise.

Phase G: Promotion

  1. Run full realistic cold suite.
  2. Run canaries at scale.
  3. Run long-context/service guard if config touches prompt processing.
  4. Repeat if the failure mode is intermittent.
  5. Update result packet, repro, and indexes.
  6. Submit to LocalMaxxing only if it is a true valid record.
  7. Commit and push.

Phase H: Closeout Or Next Model

  1. Mark exhausted ideas.
  2. Preserve patches and failures.
  3. Write “what worked” and “what did not” sections.
  4. Update docs/model-effort-index.md.
  5. Ensure the workspace is clean.
  6. Move to the next model from a known-good main.

16. Prompts For Another AI

New Model Kickoff

Start a new optimization lane for <model> on <hardware>. First read
AGENTS.md, CURRENT.md, docs/model-effort-index.md, docs/model-optimization-guide.md,
and the nearest existing result packet. Create a model packet under results/
with validity gates, runtime plan, model options, and an initial reproduce
file. Do not change runtime code until the model identity, quality lane, and
baseline benchmark are explicit.

Artifact Audit

Audit this model lane for stale claims, missing links, invalid headline
throughput, untracked useful artifacts, and oversized or secret files. Return
exact paths and whether each should be tracked, ignored, summarized, or deleted.

Experiment Proposal

Given the current valid baseline and failed paths, propose the next 12
experiments. Rank them by expected value. For each, state hypothesis, exact
config or patch, validation command, expected win size, expected failure
signature, and whether it needs four-GPU A/B or a solo confirmation.

Negative Result Summary

Turn this failed run into a reusable negative result. Include the reason we
tried it, exact identity, patch/config delta, artifacts, measured speed,
correctness status, failure signature, what class of ideas it rules out, and
what should be tried next.

Promotion Review

Review this candidate as if it is about to be submitted to LocalMaxxing. Check
fresh-response validity, cached_tokens, prompt uniqueness, quality lane,
quantization, speculative verification, metric definition, variance, server
logs, and reproducibility. Return blockers first.

17. Common High-Value Search Areas

For MoE models:

For attention-heavy models:

For speculation:

For tensor parallel or multi-GPU:

For runtime reliability:

18. Final Rule

The goal is not to get the largest number. The goal is to get the largest number that is true, quality-preserving, reproducible, and useful to the next person.

This site is open source. Improve this page.