b70-optimization-lab

2026-05-07 Q4_0 Four-Card Assist Refresh

Context

After fixing the over-broad Q8-cache guard that disabled the validated allreduce+ADD path, I reran the best known four-card Q4_0 assist split. Earlier, the four-card assist layout reached 39.204149 tok/s; equal four-card split was only 34.929313 tok/s. The question was whether the current fused stack could make the assist layout useful again without changing model quality.

Configuration

Environment:

GGML_SYCL_DISABLE_DNN=1
GGML_SYCL_Q8_CACHE=1
GGML_SYCL_ASYNC_CPY_TENSOR=0
GGML_SYCL_ASYNC_PEER_COPY=1
GGML_SYCL_COMM_ALLREDUCE=1
GGML_SYCL_COMM_SINGLE_KERNEL=1
GGML_SYCL_COMM_EVENT_BARRIER=1
GGML_SYCL_COMM_SYNC_AFTER=2
GGML_META_FUSE_ALLREDUCE_ADD=1
GGML_META_FUSE_ALLREDUCE_GET_ROWS=1
GGML_SYCL_FUSE_MMVQ2=1
GGML_SYCL_FUSE_MMVQ2_SWIGLU=1
GGML_SYCL_FUSE_RMS_NORM_MUL=1
GGML_SYCL_REORDER_MMVQ_SUBGROUPS_RUNTIME=2

Results

Run Prompt Output Repeats tok/s prompt tok/s output tok/s total
Short screen 0 128 2 n/a 42.782880 n/a
Full validation 512 512 3 117.882824 44.087560 64.174276

The full validation is:

LocalMaxxing

The detailed annotated payload returned HTTP 500, matching the previous llama.cpp detailed-payload issue. The reduced payload was accepted:

Decision

This is the new best submitted four-card Q4_0 result. It proves the current fused stack and guard fix make the assist split viable again, but it still trails TP3. Four-card Q4_0 remains an investigation path; the next useful work is reducing communication and launch overhead rather than moving more rows onto the fourth GPU.

Split Sweep

I screened nearby fourth-card weights with p0/n128/r2:

Split tok/s out
1/1/1/0.02 42.819726
1/1/1/0.05 42.785891
1/1/1/0.08 43.058166
1/1/1/0.12 42.159686
1/1/1/0.20 41.091157

The short-screen winner, 1/1/1/0.08, did not beat the submitted 0.05 result on a full validation: p512/n512/r3 produced prompt 117.366467 tok/s, decode 43.929404 tok/s, computed total 63.930204 tok/s. Keep 1/1/1/0.05 as the validated four-card config.

Subgroup Runtime Sweep

I also screened GGML_SYCL_REORDER_MMVQ_SUBGROUPS_RUNTIME on the accepted split:

Runtime tok/s out
unset/default 42.970405
1 42.988543
2 42.384839
4 42.693139

The short-screen winner, runtime 1, failed to beat the submitted runtime 2 result on a full validation: p512/n512/r3 produced prompt 114.726410 tok/s, decode 43.721328 tok/s, computed total 63.314138 tok/s. Keep runtime 2 as the validated four-card setting for now.

Sync-After Sweep

I screened GGML_SYCL_COMM_SYNC_AFTER on the accepted split:

sync_after tok/s out
0 43.065947
1 42.750153
2 43.222829
3 43.208886
4 43.081701

2 remains the best short-screen value, with 3 effectively tied but not clearly better. Keep GGML_SYCL_COMM_SYNC_AFTER=2.

Artifacts

This site is open source. Improve this page.