b70-optimization-lab

Qwen3.6 27B Q4_0 GGUF SYCL Single-Kernel Allreduce Results

Date: 2026-05-04 UTC

Host: Ubuntu 24.04.4 LTS, AMD EPYC 9015, 4x Intel Arc Pro B70 / BMG-G31 32 GB. No GPU power-limit or clock changes were made.

Model: /home/steve/models/qwen3.6-27b-q4_0-gguf/Qwen3.6-27B-Q4_0.gguf

llama.cpp worktree: /home/steve/src/llama.cpp-q4-b70, upstream db44417 plus local experimental SYCL/Vulkan patches.

Runtime stack observed locally:

Intel Compute Runtime: 26.14.37833.4
Level Zero runtime: 1.15.37833+4
OpenCL NEO: 26.14.37833.4

Key Patches

GGML_SYCL_ASYNC_CPY_TENSOR=1: non-blocking SYCL backend tensor copy hook for Meta tensor-parallel activation exchange.
Qwen recurrent split-anchor fix: recurrent attn_qkv.* and attn_gate.weight anchor to ssm_out.weight so 3-way tensor split uses a consistent split plan.
GGML_SYCL_COMM_ALLREDUCE=1: SYCL backend Meta comm hook for contiguous F32 allreduces.
GGML_SYCL_COMM_SINGLE_KERNEL=1: experimental 2-4 GPU root-kernel allreduce. Backend 0 peer-reads all partial F32 tensors and writes the summed hidden state back to every participating device.
Vulkan side patch: recognize B70 PCI ID 0xE223 as 32 Xe2 shader cores and expose GGML_VK_INTEL_XE2_DMMV_LARGE_MAX_M for DMMV experiments.

The single-kernel allreduce remains env-gated because it is more aggressive than device-to-device copy plus local sum.

Correctness Probe

Local stress test: /home/steve/sycl-peer-read-test.cpp.

Validation command:

set +u
source /opt/intel/oneapi/setvars.sh --force >/dev/null
set -u
icpx -O2 -fsycl /home/steve/sycl-peer-read-test.cpp -o /home/steve/sycl-peer-read-test
ONEAPI_DEVICE_SELECTOR=level_zero:0,1,2,3 timeout 3m /home/steve/sycl-peer-read-test

Result:

peer kernel read ok across 4 devices

The test writes all source tensors from device kernels, runs the same root-device peer-read/peer-write sum pattern used by the experimental allreduce, and verifies every device sees the exact expected F32 sum over 200 repeats.

Current Best Commands

2 B70s

ONEAPI_DEVICE_SELECTOR=level_zero:0,3 \
GGML_SYCL_DISABLE_GRAPH=0 \
GGML_SYCL_DISABLE_DNN=1 \
GGML_SYCL_DISABLE_OPT=0 \
GGML_SYCL_PRIORITIZE_DMMV=0 \
GGML_SYCL_ASYNC_PEER_COPY=0 \
GGML_SYCL_ASYNC_CPY_TENSOR=1 \
GGML_SYCL_COMM_ALLREDUCE=1 \
GGML_SYCL_COMM_SINGLE_KERNEL=1 \
/home/steve/src/llama.cpp-q4-b70/build-sycl-2026-bmg-g31/bin/llama-bench \
  -m /home/steve/models/qwen3.6-27b-q4_0-gguf/Qwen3.6-27B-Q4_0.gguf \
  --prio 0 -dev SYCL0/SYCL1 -ngl 99 -p 0 -n 512 \
  -sm tensor -ts 1/1 -b 512 -ub 32 -ctk f16 -ctv f16 \
  -t 8 --poll 50 -fa 1 -r 3 -o jsonl

Result: 39.849 tok/s, samples 39.8517, 39.7809, 39.9148.

LocalMaxxing ID: cmoqp6jpq0004lb04241n9ns3.

3 B70s

ONEAPI_DEVICE_SELECTOR=level_zero:2,1,3 \
GGML_SYCL_DISABLE_GRAPH=0 \
GGML_SYCL_DISABLE_DNN=1 \
GGML_SYCL_DISABLE_OPT=0 \
GGML_SYCL_PRIORITIZE_DMMV=0 \
GGML_SYCL_ASYNC_PEER_COPY=0 \
GGML_SYCL_ASYNC_CPY_TENSOR=1 \
GGML_SYCL_COMM_ALLREDUCE=1 \
GGML_SYCL_COMM_SINGLE_KERNEL=1 \
/home/steve/src/llama.cpp-q4-b70/build-sycl-2026-bmg-g31/bin/llama-bench \
  -m /home/steve/models/qwen3.6-27b-q4_0-gguf/Qwen3.6-27B-Q4_0.gguf \
  --prio 0 -dev SYCL0/SYCL1/SYCL2 -ngl 99 -p 0 -n 512 \
  -sm tensor -ts 1/1/1 -b 512 -ub 32 -ctk f16 -ctv f16 \
  -t 8 --poll 50 -fa 1 -r 3 -o jsonl

Result: 41.737 tok/s, samples 41.6977, 41.6966, 41.8155.

LocalMaxxing ID: cmoqqed6s0007jv049wnizwle.

Results Summary

Setup	Selector	Flags	512-token tok/s	Notes
1x B70 baseline	`0`	single device	`24.723`	Q4_0, f16 KV
2x B70 async copy	`0,3`	`ASYNC_CPY_TENSOR=1`	`37.690`	LocalMaxxing `cmoqkcqpv0006la04l5mtlj2q`
2x B70 direct allreduce	`0,3`	`COMM_ALLREDUCE=1`	`38.621`	LocalMaxxing `cmoqnkcx10006kw04f2jmahrv`
2x B70 single-kernel allreduce	`0,3`	`COMM_SINGLE_KERNEL=1`	`39.849`	LocalMaxxing `cmoqp6jpq0004lb04241n9ns3`
3x B70 split-anchor only	`0,1,2`	async copy, split fix	`38.365`	LocalMaxxing `cmoqli4dm0005l404kdf9ofnd`
3x B70 single-kernel allreduce	`0,1,2`	root 0	`41.367`	LocalMaxxing `cmoqptj6i000blb04j0i2u9yo`
3x B70 single-kernel allreduce	`2,1,3`	root 2	`41.737`	Current best, LocalMaxxing `cmoqqed6s0007jv049wnizwle`
4x B70 single-kernel allreduce	`0,1,2,3`	root 0	`31.482`	Still not viable for single-session speed

Follow-Up Sweeps

3-GPU root-order sweep, 128 tokens:

2,1,3: 41.358 tok/s
0,1,2: 41.129 tok/s
1,2,3: 41.075 tok/s
3,0,2: 41.032 tok/s

3-GPU -ub sweep:

Short-run -ub 128 won, but full 512-token validation fell to 41.113 tok/s.
Keep -ub 32.

3-GPU --poll sweep:

--poll 50 remains best.

4-GPU root-order sweep:

Best root order 3,0,1,2: 31.311 tok/s at 128 tokens.
All tested root orders stayed near 30.76-31.31 tok/s, so 4-way weakness is communication fanout/synchronization, not root-card choice.

Single-Card Follow-Up

Focused single-card flag sweep:

Output summary: /home/steve/bench-results/qwen36-q4_0-gguf/sycl-single-selector2-dnn-fa-ub-sweep-n256-20260504T052313Z.tsv.
Shape: selector 2, -dev SYCL0, -sm none, 256 generated tokens, f16 KV, no speculative decode.
Best short result: oneDNN enabled, flash attention disabled, -ub 128: 24.449 tok/s.
Flash attention was slightly slower in this decode-only shape.
oneDNN on/off and -ub 32/64/128/256 did not move the result meaningfully.

Build comparison:

Output summary: /home/steve/bench-results/qwen36-q4_0-gguf/sycl-single-build-compare-n256-20260504T053253Z.tsv.
aot-dnn: 24.805 tok/s.
dnn: 24.756 tok/s.
aot: 24.561 tok/s.
current base build: 24.498 tok/s.

Conclusion: the Linux single-card Q4_0 path is still below the Windows >27 tok/s target. The gap is not explained by oneDNN, flash attention, ubatch, or the current AOT build variants; next single-card work should profile the Q4_0 reordered MMVQ/matvec path and driver/runtime scheduling.

Current Read

The best quality-preserving GGUF/SYCL single-session setup is three B70s, equal tensor split, selector order 2,1,3, -ub 32, --poll 50, and the env-gated single-kernel allreduce path.

The fourth B70 is currently better used for a parallel session, a draft/speculative model, or further communication R&D. The remaining blocker is not raw model quality or power; it is the 128 small F32 allreduces per generated token and their synchronization cost.

This site is open source. Improve this page.