What we measured on one Intel Arc Pro B70 (32 GB) and what to use as a result. Every number links to where it was measured. Setup unless a row says otherwise: Fedora 43, kernel 7.1.8, oneAPI 2025.3, Mesa 25.3.6, one request at a time. Measured 2026-10-14/15.
Back to the hub · Leaderboard (user submissions) · Prebuilt llama.cpp
| you want | use | measured |
|---|---|---|
| Qwen3.6-35B-A3B, one user, fastest | vLLM + cookbook image ghcr.io/riceharvest/b70-vllm, MTP k=4 |
243.7 code / 164.4 prose tok/s (cookbook) |
| Qwen3.6-35B-A3B in llama.cpp | Vulkan b70 build + MTP n=3 | 108.5 tok/s, output identical to no MTP (test 001) |
| Qwen3.6-35B-A3B, 64 users | vLLM + --mamba-ssm-cache-dtype float16 |
1420-1511 generation, 2581-2769 total tok/s (below) |
| Qwen3.8-27B, one user | vLLM + cookbook | code 113 / prose 66 tok/s, short prompt; 72 at 32K (cookbook) |
| Qwen3.8-27B in llama.cpp | Vulkan b70 build on Mesa 26.2.5 | 25.3 tok/s (SYCL 21.9) (test 002) |
| long prompts in llama.cpp | SYCL | prompt after 32K context: 720 vs 86 tok/s on Vulkan (27B, Mesa 25) (hub table) |
| NVIDIA Nemotron 3.5 Bolt 74B-A9B (vision) | llama.cpp master Vulkan on Mesa 26.2, NVIDIA’s 24 GB GGUF + LiLiCorr drafter | 43.2 tok/s decode, 54.4 on code with the drafter, 81.6 total with 4 users (below) |
| Nemotron 3.5 Bolt closer to BF16, card + 32 GB+ RAM | our Q4_K_M from BF16, llama.cpp b11669+, experts partly in RAM | KL 0.015 vs 0.169 for NVIDIA’s 2-bit, PPL 4.56 vs 5.28; B70 speed pending (below) |
| no oneAPI on the machine | Vulkan, with Mesa 26 | Mesa 26 more than doubles MoE decode vs 25 (below) |
Rule of thumb: on Mesa 25.x use llama.cpp SYCL for speed; on Mesa 26 Vulkan is as fast or faster on decode. For MTP in llama.cpp, Vulkan is the safe choice today (exact output); SYCL MTP on master is slower than no MTP.
llama.cpp Vulkan, same binaries, only the Mesa driver changed. llama-bench -fa 1 -r 3, tok/s (test 002).
| model / test | master, Mesa 25.3.6 | master, Mesa 26.2.5 | b70 build, Mesa 25.3.6 | b70 build, Mesa 26.2.5 | SYCL |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B UD-Q4_K_M decode | 39.0 | 86.1 | 64.3 | 87.3 | 97.7 |
| Qwen3.6-35B-A3B UD-Q4_K_M prompt (512) | 1854 | 1637 | 1931 | 1737 | 1301 |
| Qwen3.8-27B decode | 13.2 | 22.9 | 22.7 | 25.3 | 21.9 |
Mesa 26 costs ~10% on short prompts. “b70 build” is the Vulkan tarball from
b70-llama-builds (branch vk-b70-stack).
Qwen3.6-35B-A3B UD-Q4_K_M, Mesa 25.3.6, tok/s (branch doc):
| test | Vulkan master | Vulkan b70 build | SYCL master |
|---|---|---|---|
| decode | 39.0 | 64.3 | 97.7 |
| decode after 32K | 32.7 | 52.5 | 74.5 |
| prompt (512) | 1854 | 1931 | 1301 |
| prompt after 32K | 312 | 630 | 1083 |
Dense 27B and a 1.7B model: hub table.
Qwen3.6-35B-A3B-MTP UD-Q4_K_M, llama-server, greedy, 3 prompts x 256 tokens, mean tok/s
(test 001, test 007):
| backend | no MTP | MTP n=1 | n=2 | n=3 | output vs no MTP |
|---|---|---|---|---|---|
| Vulkan b70 build | 70.7 | 91.3 | 100.8 | 108.5 | identical |
| SYCL master | 86.7 | 69.0 | 77.0 | 80.9 | differs |
SYCL, fork branch sycl-mmid-small-batch (not upstream) |
83.9 | 131.8 | differs (being checked) |
Why SYCL master is slow with MTP: its MoE path for 2-8 token verify batches goes through a host-synced sort and grouped GEMM; the fork branch keeps those batches on the single-token kernel.
A port of the public-domain qwen35moe fusions from steveseguin’s b70-optimization-lab onto llama.cpp master, each behind an environment flag (test 006, branch doc):
| Qwen3.6-35B-A3B UD-Q4_K_M, tg128 | tok/s |
|---|---|
| SYCL master | 93.4 |
| fork branch, fusions on | 107.9 (+15.5%), greedy output identical |
Not upstream and not in the prebuilt builds yet.
NVIDIA’s Mamba-2/Transformer LatentMoE (74B, 9B active, vision), the
24 GB GGUF (2-bit QAT), llama.cpp master
620d2225c (Bolt support: #30520), Vulkan (details):
| Mesa 26.2.5 | Mesa 25.3.6 | |
|---|---|---|
| decode (tg128) at 0 / 8K / 32K context | 43.2 / 41.7 / 39.0 | 25.0 / 24.9 / 23.3 |
| prompt (pp512) at 0 / 8K / 32K | 680 / 526 / 326 | 768 / 606 / 303 |
| llama-server, code / prose answers | 42.8 / 42.4 | 27.3 / 27.3 |
+ LiLiCorr drafter (--spec-draft-n-max 6) |
54.4 / 42.5 | 32.3 / 23.5 |
| 4 users at once, total | 81.6 | 54.5 |
Mesa 26 is worth ~70% here. Vision works with the mmproj on the CPU (~5 s per image). With the drafter at 32K context the card uses 22.9 GiB. SYCL + drafter currently aborts (see known problems).
Our 3- to 8-bit ladder, quantized from
NVIDIA’s BF16 checkpoint (quality, memory and method). On the Hub now: Q4_K_M.
Coming: IQ3_XXS, Q3_K_M, IQ4_XS, Q5_K_M, Q6_K, Q8_0. Needs llama.cpp b11669+ (Ollama 0.40.5+, untested, single-file
GGUFs only for hf.co/ pulls; LM Studio unverified; the b70-llama-builds release is not on b11669 yet, see
known problems). B70 speed: pending.
Which file: the experts that don’t fit in VRAM stay in system RAM (-ncmoe). Weights + 32K context + buffers,
computed, not measured (table):
| file | size GiB | 24 GB card | 32 GB card (B70) |
|---|---|---|---|
| NVIDIA 24 GB (QAT IQ2) | 20.5 | fits whole | fits whole |
| IQ3_XXS (coming) | 29.8 | ~21 GiB VRAM + ~10 GiB RAM | ~29 + ~3 |
| Q3_K_M (coming) | 36.1 | ~21 + ~17 | — |
| IQ4_XS (coming) | 39.1 | ~21 + ~20 | — |
| Q4_K_M | 45.0 | ~21 + ~26 (-ncmoe 64) |
~28 + ~18 (-ncmoe 46) |
Quality: llama-perplexity, wikitext-2 test, 16 x 2048 tokens; KL and same top-1 token vs our Q8_0 (details):
| file | size GiB | PPL | KL mean | KL 99th pct | same top-1 |
|---|---|---|---|---|---|
| NVIDIA 24 GB (QAT IQ2) | 20.5 | 5.284 ± 0.096 | 0.169 | 1.59 | 84.2% |
| Q4_K_M (imatrix) | 45.0 | 4.563 ± 0.077 | 0.0150 | 0.117 | 94.1% |
| Q4_K_M, no imatrix | 45.0 | 4.566 ± 0.077 | 0.0151 | 0.118 | 94.2% |
NVIDIA’s file comes from its own QAT-retrained checkpoint, so its KL vs our BF16-based Q8_0 includes the retraining, not only rounding; PPL compares both on equal terms. imatrix vs no imatrix is within noise.
palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4, 16K context, code / prose answers of 512 tokens, decode tok/s (cookbook):
| setup | no MTP | MTP k=4 |
|---|---|---|
| stock vLLM XPU v0.31 image | 75.6 / 76.3 | 123.4 / 113.2 (fits only at 2K context) |
| + cookbook patches | 73.4 / 73.7 | 175.9 / 126.6 |
| + fp8 lm_head | 78.7 / 79.1 | 183.9 / 134.3 |
| + int4 MoE decode GEMV | 93.8 / 94.3 | 200.8 / 145.4 |
| + fp8 attention/GDN/shared expert (image default) | 118.9 / 119.6 | 243.7 / 164.4 |
Two of these are proposed upstream, measured on vLLM nightly (4K context, test 004, test 005):
| change | before | after |
|---|---|---|
| int4 MoE decode GEMV (vllm-xpu-kernels #644), no MTP | 75.8 / 76.3 | 86.6 / 87.2 |
| + fp8 lm_head via online quantization (vllm #61815), no MTP | 86.6 / 87.2 | 89.4 / 90.2 |
| same two, MTP k=4 (2K context) | 157.3 / 118.7 | 183.7 / 131.9 |
gsm8k (first 100) stayed at 96-97/100 in every arm.
Qwen3.6-35B-A3B, 64 requests in flight, 512-token prompts and answers, no speculative decoding (details, quality check and the exact command):
| setup | running at once | median time per token | generation (64 / TPOT) | total tok/s |
|---|---|---|---|---|
| vLLM default (fp32 GDN state) | 61 of 64 | 49.3 ms | 1298.7 | 2092 |
--mamba-ssm-cache-dtype float16 |
64 of 64 | 42.3-45.1 ms | 1420-1511 | 2581-2769 |
The fp16 state halves the per-request linear-attention state (fp32 by default for Qwen3.5/3.6/3.8); decode quality stays at the fp32 noise floor in our check (gsm8k 98/100 vs 94/100 for fp32). Two runs: 1511.5 and 1420 generation. Best 64-user entry on localmaxxing for this model on one B70: 1139.8.
--prebuilt downloads the B70 builds,
so one command benchmarks SYCL and Vulkan and offers to add your result to the shared table:
curl -fsSL https://raw.githubusercontent.com/riceharvest/b70-bench/main/b70-bench.sh | bash -s -- --prebuilt --tier quick