Cross-model study · 20 September 2026

CUDA prefill across six MoE models

Six mixture-of-experts models, two backends and four prompt lengths. This study measures how CUDA prompt processing compares with Metal as active parameter count and context length change.

6 models

the same expert-residency policy across the study

3–39B

active parameters per token

12k–48k

tokens across four prompt lengths

10.37×

highest measured prefill speedup · Mixtral at 48k

Measured prefill results

Metal prefill time divided by RTX 5090 prefill time. Values above 1× favour CUDA. State transfer and token generation are outside this comparison.

The 12k runs are cold starts. They are the first prefill after loading each model and include loading expert data from disk. The later runs are warm; use those to compare context lengths.

Scroll the table to view all columns

CUDA prefill speedup relative to Metal, by model and prompt length
Model	Active	12kcold start	24k	26k	48kwarm	Expert bytes	24k → 26k
Qwen3-Coder-Next80B-A3B	3B	0.68×	2.77×	2.09×	3.86×	43.7 GiB	6.9 s
gpt-oss-120b117B-A5.1B	5.1B	0.68×	3.14×	2.13×	4.24×	56.9 GiB	9.8 s
Qwen3.5 122BA10B	10B	0.97×	4.29×	3.23×	5.47×	65.4 GiB	11.5 s
DeepSeek-V4-FlashMLA, ~A10B	~10B	1.28×	3.22×	2.46×	3.07×	83.6 GiB	16.1 s
Llama-4-Scout109B-A17B	17B	1.66×	5.16×	3.89×	5.40×	54.5 GiB	9.9 s
Mixtral 8x22B141B-A39B	39B	3.33×	9.90×	6.76×	10.37×	71.5 GiB	14.5 s

Measured 20 September 2026, 01:23–02:44. Both backends received identical token IDs for each model. “24k → 26k” is the increase in CUDA elapsed time across the microbatch boundary, including the extra 2,000 tokens of work. Expert data sizes are in GiB.

Every model ran with all experts in host memory. This keeps the residency policy consistent; approximately 20 GiB of card memory remained available for resident experts. Per-model residency tuning was not measured, so these results do not quantify its potential benefit.

What the results show

Higher active-parameter counts tend to favour CUDA

At 24k tokens, the measured speedup ranges from 2.77× for Qwen3-Coder-Next to 9.90× for Mixtral. Active weight data per token spans 3.4, 3.9, 5.8, 8.5, 9.8 and 21.3 GiB across the six models. The broad trend supports the compute hypothesis, but the ordering is not strictly monotonic and total model size alone does not explain the results.

Architecture matters

DeepSeek is the exception to increasing gains at longer context. Its speedup falls from 3.22× at 24k to 3.07× at 48k. Metal throughput remains approximately 178 tok/s out to 48k in the study notes. Its multi-head latent attention (MLA) architecture is a possible explanation; this comparison does not isolate that effect.

Crossing a microbatch boundary adds a measurable cost

The 24k-to-26k step adds 6.9–16.1 seconds across expert sets of 43.7–83.6 GiB. That increase is consistent with another transfer of expert data, alongside the extra computation. Dividing expert bytes by the elapsed step gives an implied rate of approximately 4.9–6.4 GiB/s; it is not an isolated measurement of link bandwidth.

The lower implied rates on larger models may reflect page-cache pressure and additional disk reads. That explanation remains untested.

The cold-start point limits the cost-model fit

A per-model fit of ⌈tokens ÷ 24,576⌉ × transfer + tokens ÷ rate + fixed produced residuals of 3.5–6.6 seconds and an infinite marginal rate for two models. The cold first point distorted the fitted constant, so those parameter estimates are omitted. The table reports the observed 24k-to-26k timing difference instead.

Model selection

The set spans active parameter counts, expert data sizes and attention architectures.

Scroll the table to view all columns

Model files, active parameters, estimated KV cache and study roles
Model	Size	Active	KV / token	Role in the study
Qwen3-Coder-Next 80B-A3BUD-Q4_K_XL · qwen3next	49.6 GB	3B	96 KB	Baseline. Connects the study to the existing cost model and earlier 24k- and 48k-token measurements.
gpt-oss-120bMXFP4 · gpt-oss	63.4 GB	5.1B	72 KB	A second model with relatively few active parameters; the comparison uses the study’s own Metal measurements.
Qwen3.5 122B-A10BQ4_K_M · qwen35moe	69.1 GB	10B	96 KB	An intermediate active-parameter count, selecting 8 of 256 experts per token.
DeepSeek-V4-FlashUD-Q2_K_XL · deepseek4	96.8 GB	~10B	86 KB	A similar active-parameter count to Qwen3.5, with MLA attention. Quantisation, file size and kernels also differ.
Llama-4-Scout 109B-A17BQ4_K_M · llama4	65.4 GB	17B	192 KB	A higher active-parameter count and the second-largest estimated KV cache per token in the set.
Mixtral 8x22BQ4_K_S · llama	80.5 GB	39B	224 KB	The largest active-parameter count and estimated KV cache per token; tests compute and context-length sensitivity.

Active parameters are in billions (B); file sizes are in decimal GB. KV-cache estimates use model-header dimensions and two bytes per cached value: layers × KV heads × (key dimension + value dimension) × 2. Architecture-specific layouts may differ. The DeepSeek estimate requires runtime confirmation. All model files were stored on the same external drive.

Test protocol

CUDA backendRTX 5090 through macuda over Thunderbolt

Metal backendThe same Mac’s GPU, using the same model file

Prompt lengths12,000 · 24,000 · 26,000 · 48,000 tokens

Expert residencyAll experts in host memory; no per-model tuning

Match token counts and backend inputs

Each model tokenizes the same source text with its own tokenizer, then the sequence is truncated to the target length. CUDA and Metal receive identical token IDs for that model. Token counts match across models even when their tokenizers differ.

The 24k and 26k targets sit on opposite sides of the 24,576-token microbatch boundary, where expert data must be transferred again. The 12k and 48k runs extend the context range.

Cache and residency controls

The prompt cache is cleared between lengths to prevent shared-prefix reuse. All runs use --n-cpu-moe n_layer, matching the initial cost model’s all-experts-on-host configuration. Clearing the prompt cache does not clear the operating system’s file cache.

Hypotheses and interpretation limits

The hypotheses were recorded before measurement. Their outcomes are summarised against the data above.

Hypothesis 1 · broadly supported

More active parameters produce larger prefill gains

The predicted ordering placed Mixtral at the high end and Qwen3-Coder-Next at the low end. The 24k results follow that broad pattern, with architectural exceptions. Transfer volume, quantisation and kernel differences prevent attributing the gains to active parameters alone.

Hypothesis 2 · not universal

Relative prefill gains increase with prompt length

The earlier Qwen3-Coder-Next measurements rose from 2.35× near 24k to 5.55× near 48k. In this sweep, five models improve from 24k to 48k, but DeepSeek falls from 3.22× to 3.07×. That result refutes the universal claim.

Hypothesis 3 · consistent with the measurements

A second microbatch requires another expert transfer

Every model takes longer at 26k than at 24k, and the increases broadly track expert-set size. The observed steps support the proposed transfer mechanism, though the timing difference also includes additional computation.

KV-cache effects remain unresolved

Estimated cache size, active parameter count, attention architecture and quantisation vary together. These context sweeps describe model behaviour but do not isolate its cause. Because all experts remain on the host, this study does not test the earlier idea that KV-cache growth displaces resident experts.

DeepSeek’s cache layout still needs confirmation. The 86 KB/token estimate assumes an expanded cache. A compressed 512-wide latent plus rotary-position data would imply approximately 48 KB/token. The server’s reported KV-buffer allocation at a known context length is needed to resolve the difference.

Stability during the sweep

A completed observation window of 1 hour 21 minutes, measured from the first card prefill. These figures are a recorded snapshot.

1 h 21 min

unbroken observation window under load

card prefills across six models

recorded DART / IOMMU faults

re-enumerations under load

tools/card-uptime.sh samples process information, the I/O registry and the lock file every five minutes without opening the device or creating a second hardware client.

What the observation window establishes

The reported duration covers the sweep’s work on the card. Extension process lifetime is used as an invalidation signal: macOS restarts the extension when the device is re-enumerated. An unbroken instance is interpreted alongside device-enumeration and error logs, rather than used alone as evidence of stability.

Baseline event · resolved

The host rebooted at 21:45:12 on 19 September, attributed in the study notes to a faulty USB device. The driver extension started 28 seconds later, at 21:45:40. Preflight detected the changed instance and was cleared manually before the sweep. The host boot time and extension start time explain the change in instance.

The notes report a 16 GT/s ×4 link, no latched DART errors and no abnormal Thunderbolt events. The extension’s age therefore reflects host uptime; the duration above begins with the sweep’s first card step.