Files: darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF. On the Hub now: Q4_K_M (45.0 GiB, 2 parts) and the vision projector (mmproj Q8_0 / BF16). Coming, each after its quality check: IQ3_XXS, Q3_K_M, IQ4_XS, Q5_K_M, Q6_K, Q8_0.
Measured 2026-10-15. B70 speed: pending (only NVIDIA’s 24 GB file has B70 numbers so far, in bolt-24gb-b70.md).
llama-perplexity on wikitext-2 test, 16 windows of 2048 tokens (32,768 tokens). KL divergence and same top-1 token
are against our Q8_0, because the BF16 model (138 GiB) does not fit the test machine; Q8_0 is normally within noise of BF16.
| file | size GiB | PPL | KL vs Q8_0 (mean) | KL 99th pct | same top-1 token |
|---|---|---|---|---|---|
| Q8_0 (reference, upload coming) | 73.4 | 4.541 | reference | ||
| Q4_K_M (imatrix, on the Hub) | 45.0 | 4.563 ± 0.077 | 0.0150 | 0.117 | 94.1% |
| Q4_K_M without imatrix (not published) | 45.0 | 4.566 ± 0.077 | 0.0151 | 0.118 | 94.2% |
| NVIDIA 24 GB (QAT IQ2) | 20.5 | 5.284 ± 0.096 | 0.169 | 1.59 | 84.2% |
Caveat: NVIDIA’s file is quantized from its own QAT-retrained checkpoint, so its KL against our BF16-based Q8_0 includes that retraining, not only rounding error. PPL compares both files on equal terms.
The imatrix and no-imatrix Q4_K_M are the same within noise; the published file uses the imatrix.
Weights + 32K context + compute buffers, from the file sizes. The routed experts are ~89% of the weights, so a file
bigger than the card runs by keeping some experts in system RAM (-ncmoe N, or let llama.cpp’s default --fit on pick).
| file | size GiB | status | 24 GB card | 32 GB card (B70) |
|---|---|---|---|---|
| NVIDIA 24 GB (QAT IQ2) | 20.5 | on the Hub (NVIDIA) | fits whole | fits whole (measured on the B70: 22.9 GiB with the LiLiCorr drafter at 32K) |
| IQ3_XXS | 29.8 | coming | ~21 GiB VRAM + ~10 GiB RAM | ~29 GiB VRAM + ~3 GiB RAM |
| Q3_K_M | 36.1 | coming | ~21 + ~17 | not computed yet |
| IQ4_XS | 39.1 | coming | ~21 + ~20 | not computed yet |
| Q4_K_M | 45.0 | on the Hub | ~21 + ~26 (-ngl 99 -ncmoe 64) |
~28 + ~18 (-ngl 99 -ncmoe 46) |
| Q5_K_M, Q6_K, Q8_0 | coming |
Measured VRAM replaces these numbers after the B70 runs.
convert_hf_to_gguf.py plus the Bolt fix on
riceharvest/llama.cpp bolt-convert
(master’s converter fails on this checkpoint, see known problems).llama-quantize with per-tensor overrides: routed experts take the file’s type; attention and the LatentMoE
projections stay Q8_0, Mamba in/out and the shared expert Q6_K (for Q4_K_M). Full table and scripts are on the
model card.llama-perplexity --kl-divergence against the Q8_0 logits, numbers as above.Needs llama.cpp b11669+ (#30520). Ollama needs
0.40.5+ (bundles b11669; not tested by us), and ollama run hf.co/...
only pulls files stored as one GGUF: the Hugging Face registry refuses split (-0000N-of-) files. LM Studio: not verified. The b70-llama-builds release
is not on b11669 yet (a Bolt update is waiting for a GPU test); build upstream for now (see
bolt-24gb-b70.md).
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF --include "Q4_K_M/*" --local-dir bolt
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf --local-dir bolt
./build/bin/llama-server -m bolt/Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf \
--mmproj bolt/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf --no-mmproj-offload \
-ngl 99 -ncmoe 46 -fa on -c 32768 --jinja
-ncmoe 46 is the computed split for a 32 GB card; use 64 on a 24 GB card, or drop -ngl/-ncmoe and let --fit choose.