Record: MI25 re-benchmark on llama.cpp 11193 — 2026-09-25

Outcome

Re-measured the MI25 (Vulkan, RADV VEGA10) on the upgraded llama.cpp build and compared against the records/2026-09-21-01-mi25-inference-benchmarks.md baseline, which was taken on 0.4.0-dev build 10893.

Build 11193 is safe to keep: no resident model regressed, and gemma-12B prefill at full offload improved by about 45%. A new ngl sweep on Qwen3.8-Flash-Next shows that partial offload is strictly worse at every step, which is stronger evidence than the two endpoints the earlier record measured.

Environment (verified 2026-09-25)

• The owner-reported tag is b11192; the binaries report build 11193.
• llama.service and llama-cpu.service are active with NRestarts=0, both started 2026-09-25 17:59.
• Both builds still default to n_threads = 16, n_threads_batch = 16 on beast’s 32 threads, so the threads-batch improvement remains untested.
• scripts/preset-keys-check.sh was run against the new binary before relying on it: 44 keys OK on beast and 42 OK on athena, with zero BAD keys. The 0.4.0 upgrade broke presets on a renamed key, so this check is now routine for every rebuild.

Resident models: baseline versus 11193

Same tests, models, and flags as the 2026-09-21 baseline. Generation is tg128; prefill is pp512.

The gemma-12B full-offload prefill gain is the only material change. It was re-measured three times to separate signal from noise: pp512 of 342.91, 340.26, and 339.92, against tg128 of 29.72, 30.73, and 30.69. Generation is unchanged for both models. The gemma-12B ngl 0 figures are the least trustworthy cells in the table; the caveats below explain why.

Qwen3.8-Flash-Next partial-offload sweep

batiai/Qwen3.8-Flash-Next-GGUF:Q4_K_M, reported as qwen4exp A3B Q4_K - Medium, 110.96 GiB, 176.94 B params. The earlier record measured only ngl 0 and ngl 99.

Generation falls monotonically with offload depth, and prefill collapses as soon as any layer moves to the GPU. The working set cannot fit in 15.98 GiB, so every offloaded layer adds PCIe traffic without reducing the bytes that must still be read from system RAM. Partial offload is therefore not a middle path for this model.

Supporting facts (verified 2026-09-25)

• MI25 total VRAM is 17,163,091,968 bytes (15.98 GiB), unchanged.
• The router loads models on demand per request, one child server per model id; it holds no weights itself. A request naming an unknown model returns HTTP 400 model '<id>' not found rather than falling back to a default.
• rocm-smi reported about 8.4 GB of VRAM as used while no model was loaded. This is recorded as an observation, not a finding: no leak is claimed, and the accounting is not explained.

Caveats

• The 10893 baseline could not be re-run. The matching llama-bench front-end was overwritten by the upgrade; only the 0.4.0 libraries survive in /usr/local/lib. A controlled A/B would require repointing the versioned library symlinks for live services, which was judged not worth the risk. “Change” in this record therefore means 11193 measured against numbers recorded on 2026-09-21, not a single experiment.
• The baseline numbers may have been taken with a contaminated GPU. At the time of the 2026-09-21 run, gemma-12b was resident from an unrelated check, holding roughly 10 GB of VRAM. If that is what happened, the baseline is understated, the true gemma-12B ngl 99 gain is smaller than 45%, and the ngl 0 comparison is unreliable. This is the reason to distrust the one negative figure in the resident-model table.
• Measurements use -r 1, so per-configuration variance is narrower than the between-run variance implied by the gemma-12B ngl 99 repeats. Treat single-digit deltas as noise.

Decisions

• Keep build 11193 on both hosts; no regression was found on resident models and gemma-12B prefill improved.
• Keep qwen38 at ngl = 0 if it is kept at all, and do not attempt partial offload on the MI25. The sweep shows offload helps at no depth.
• Leave the 2026-09-21 record unchanged. It remains the 10893 baseline, and this record supersedes only its currency, not its content.

Verification

• Benchmarks run 2026-09-25 from /usr/local/bin/llama-bench with GGML_VK_VISIBLE_DEVICES=1, against the GGUF files in ~/.cache/huggingface/hub/.
• Re-measure with, for example: GGML_VK_VISIBLE_DEVICES=1 llama-bench -m <model.gguf> -ngl 0,8,16,32,99 -p 512 -n 128 -r 1 --no-warmup
• The GPU was cleared of resident children before measuring, and the router reported the loaded set between steps.
• Temporary reconnaissance scripts were removed from the host and are not committed.

Commits

• See git log -- records/2026-09-25-02-mi25-rebenchmark-11193.md — docs: MI25 re-benchmark on llama.cpp 11193 (partial-offload sweep, gemma-12B prefill win); record 2026-09-25. The commit that adds this file is cited by subject and path rather than by hash, because a commit cannot contain its own hash.