Performance report · 19 September 2026

Measured performance across nine AI models

Six language models and three image models, measured on macuda and compared with the same RTX 5090 connected to a Windows PC. The language-model batch ran at 21:07–21:11; image and speculative results come from the earlier 18:12–18:16 batch. Additional serving tests are reported separately.

CardRTX 5090 (GB202) in an AORUS AI BOX over Thunderbolt, on an Apple Silicon Mac

Language-model buildlibtinynv + libtinycudart, build abd0458, defaults; CUDA graphs on, device-to-device copies by kernel, ggml-cuda host halves at -O2

Settingsllama-bench -ngl 99 -p 256 -n 128 -r 3, matching the reference settings

Native referenceSame card in a Windows PC, driver 595.79, llama.cpp b10970, byte-identical models (14 Sep)

Host loadLoad average 5.9–8.7 on a shared Mac; decode throughput varies by a few percent with host load

Validation450/450 operator checks at three chain depths, byte-identical greedy text and PNG outputs, and a five-minute server soak on 19 September

Text generation and prompt processing

Decode generates the response one token at a time. Prefill processes the input prompt. Throughput is measured in tokens per second (tok/s); higher is better. Here, “native” means the Windows reference.

Model	Size	tg128 tok/s	native	Decode vs native	pp256 tok/s	native	Prefill vs native
Llama 3.1 8B InstructQ8_0 · dense	7.95 GiB	161.0 ± 1.0	164.5	98%	8,927 ± 349	11,118	80%
Nemotron Nano 9B v2Q6_K · Mamba-hybrid (SSM)	8.50 GiB	141.5 ± 0.9	140.6	101%	6,894 ± 175	4,031	171%
Gemma 4 12B itQ4_K_M · dense	6.86 GiB	133.4 ± 1.0	136.4	98%	5,617 ± 207	4,169	135%
Gemma 4 26B-A4B itQ4_K_XL (QAT) · MoE	13.26 GiB	225.5 ± 6.5	227.8	99%	7,063 ± 207	5,478	129%
Qwen3.8 27BUD-Q4_K_M · dense	15.32 GiB	73.4 ± 0.1	75.3	97%	2,592 ± 69	2,447	106%
Qwen3.5 35B-A3BQ4_K_M · MoE	19.71 GiB	223.0 ± 1.9	245.0	91%	5,250 ± 209	4,726	111%

tg128 measures generation of 128 tokens; pp256 measures processing of a 256-token prompt. Values are the mean ± standard deviation of three repeats. Percentages are calculated from the displayed means and rounded to the nearest whole percent. The native prefill measurements have substantial variance (±2,771 tok/s for Gemma 12B and ±1,979 for Llama 8B), so those ratios should be interpreted cautiously.

Windows reference · driver 595.79macuda · 19 September 2026

Speculative decoding with a multi-token prediction head

A multi-token prediction (MTP) head proposes draft tokens for the main model to verify. Throughput depends on how many of those proposals are accepted.

Run	Tokens	Draft depth	Accepted	tok/s	Note
Qwen3.8 27B + MTP headUD-Q4_K_M + mtp Q4_0 · draft-mtp	~258	4	28–38%	86–102	llama-speculative-simple, the reference's Thunderbolt prompt, three runs: 101.7, 87.6 and 86.5 tok/s at 37.6%, 28.6% and 28.0% acceptance. Acceptance varies between runs, which affects throughput. Server results are reported separately below.

Greedy, 256 tokens, the same prompt on both sides. The reference record's own attempt at this exact test did not run (the Windows build lacked the binary); its llama-server MTP figures were 114–153 tok/s by prompt type at 44–62% acceptance. macuda's server soak recorded 120–125 tok/s with the same head. These are different test configurations, so they do not establish a direct speedup.

Image generation

Measured at 18:12–18:16 on build 05d8b4a. These timings are reported separately from the later language-model batch.

Model	Settings	Total	Sampling	Native total	Native sampling	Output
SDXL Turbofp16	512², 4 steps, cfg 1.0	2.78 s	1.55 s	4.11 s	2.45 s	PNG matches validation output byte for byte
Stable Diffusion 1.5emaonly, fp32	512², 20 steps, cfg 7	3.71 s	3.16 s	3.86 s	3.18 s	PNG matches validation output byte for byte
Z-Image TurboQ8 DiT + Qwen3-4B encoder + FLUX VAE	1024², 8 steps, cfg 1.0	10.06 s	8.41 s	9.46 s	7.18 s	PNG matches validation output byte for byte; text encoding 0.95 s, VAE decode 0.69 s. The initial run took 16.7 s with a cold encoder and VAE; the reported value is a warm rerun.

All runs use seed 42 and the reference prompt. Total time is stable-diffusion.cpp's generate_image duration; sampling measures only the diffusion loop. The Windows reference log records thermal throttling, which affects the comparison. PNG files from this batch match the project's earlier validation outputs from 19 September byte for byte; this is a check across macuda builds.

A red fox on a mossy log, generated by SDXL Turbo at 512 by 512 — **SDXL Turbo**512², 4 steps — 2.78 s

A red fox in a forest, generated by Stable Diffusion 1.5 at 512 by 512 — **Stable Diffusion 1.5**512², 20 steps — 3.71 s

A detailed photograph of a red fox on a mossy log, generated by Z-Image Turbo at 1024 by 1024 — **Z-Image Turbo**1024², 8 steps — 10.06 s

Images generated on the RTX 5090 from the Mac, resized and converted to JPEG for the website. Output comparisons were performed on the original PNG files.

Interpreting the results

Five models reach 97–101% of reference decode throughput; Qwen3.5 35B-A3B reaches 91%. An earlier Nemotron Nano optimisation replaced copy-queue transfers with copy kernels, allowing transfers to remain in the launch chain. That experiment improved throughput from 111 to 143 tok/s; the batch above records 141.5 tok/s. Profiling attributes the remaining Qwen MoE gap to idle time between chains and approximately 1.7 µs of driver overhead per launch.

The Windows reference was measured on 14 September; this macuda batch was measured on 19 September. Both used the same GPU in the same AORUS enclosure over Thunderbolt at PCIe Gen 4 ×4. The reference ran llama.cpp b10970; macuda used b10950 plus one commit. The project's source comparison found that the intervening ggml-cuda change affected AMD code paths only.

These results compare complete host-and-driver configurations. They do not isolate driver overhead from host load, thermal behaviour or run-to-run variance. Performance claims about individual macuda changes use interleaved A/B measurements on the same Mac; this table is a dated comparison with the Windows reference.

Beyond the batch

Additional serving and split-inference results from separate runs. Conditions and available dates are listed below.

Workload	This stack	Reference	Conditions
27B + MTP head, servedllama-server, single stream, greedy	124 tok/s	124.7	First request; 112 tok/s mean over a 62-minute soak with every greedy probe byte-identical.
27B + MTP, eight slotsllama-server, temp 0.6	~207 tok/s	—	Aggregate across eight concurrent slots. Twenty-minute soaks: 440 requests, 0 errors.
Qwen3-Coder-Next 80B-A3B prefill49.6 GB model, 23,692-token prompt · 16 Sep	15.2 s 1,558 tok/s	35.7 s 664 tok/s	Disaggregated: the model does not fit the card's 32 GB, so experts stream across the link once per processing microbatch, the card prefills, the 662 MB slot state is saved in 0.7 s and restored into a Metal server in 0.1 s, and Apple's own GPU decodes at 54 tok/s. The reference column here is Metal alone on the same Mac.

Validation methodology

The project uses the following checks to validate builds and evaluate performance changes:

Value-checked operator tests — test-backend-ops at 450/450, run at three different chain depths and once in synchronous mode, to check correctness across batching and synchronisation settings.
A byte-compared greedy decode — 96 tokens at temperature 0, compared byte-for-byte against a reference file. This detects output differences that can remain invisible in an operator-only test.
A verified build id. Every binary carries the driver's git commit, printed at startup and checkable with strings. This ties a result to the implementation that produced it.
Interleaved A/B for build-to-build comparisons. This Mac is shared with other work and decode speed moves several percent with host load, so the old and new builds are measured alternately in the same time window.

The companion page, Architecture, explains how the implementation behind these results, the remaining performance constraints and the corrections documented during development.