CUDA prefill.
Metal generation.
Process the prompt on an RTX 5090, then generate the response on the Mac. Split inference lets models larger than the card’s 32 GB of memory benefit from CUDA prompt processing.
One model, two processing stages
During prefill, experts stream from host memory over Thunderbolt to the RTX 5090. The conversation state then transfers to a Metal server for token generation. The model’s experts do not all need to fit in the card’s memory.
CUDA prefill
The RTX 5090 processes the prompt, streaming experts once per microbatch.
State handoff
The conversation state is saved and restored into the Metal server.
Metal generation
The Mac’s GPU generates the response using the transferred state.
The experiments below measure prefill only. State-transfer time and subsequent generation are excluded from the reported speedups.
Measured prefill gains
Qwen3-Coder-Next benefits from CUDA at longer context in both runs. The size of that benefit varies, particularly in the Metal measurements.
On 20 September, prefill speedup increased from 2.77× near 24k tokens to 3.86× near 48k. The earlier run recorded 2.35× and 5.55×. The long-context result has not replicated at the original magnitude; use the more recent 3.86× figure when citing this experiment.
Scroll the table to view all columns
| Prompt | RTX 5090 | Metal | Metal rate | Prefill speedup |
|---|---|---|---|---|
| 23,691 tokens19 Sep, first run | 15.0 s | 35.2 s | 673 tok/s | 2.35× |
| 26,156 tokens19 Sep, first run | 22.0 s | 64.6 s | 405 tok/s | 2.94× |
| 48,667 tokens19 Sep, first run | 28.9 s | 160.3 s | 304 tok/s | 5.55× |
| 23,999 tokens20 Sep, re-measured | 13.5 s | 37.5 s | 641 tok/s | 2.77× |
| 25,999 tokens20 Sep, re-measured | 20.4 s | 42.7 s | 610 tok/s | 2.09× |
| 47,999 tokens20 Sep, re-measured | 27.2 s | 104.9 s | 458 tok/s | 3.86× |
Qwen3-Coder-Next 80B-A3B, approximately 46.2 GiB (49.6 GB), with all experts in host memory. Both backends use the same model file and matching prompts within each run. The first measurements were recorded on 19 September, 21:20–21:34; the repeat comes from the 20 September cross-model study.
The long-context difference is on the Metal side
RTX 5090 timings at the short and long endpoints agree within 10% across the two runs: 15.0 versus 13.5 seconds, and 28.9 versus 27.2 seconds. Metal is also close near 24k, at 35.2 versus 37.5 seconds. Near 48k, however, Metal elapsed time falls from 160.3 to 104.9 seconds, accounting for most of the change in the reported speedup.
Storage and page-cache pressure are the leading hypothesis. The first run read the 49.6 GB model from a full external SSD measured at 41 MB/s. For the repeat, the model had moved to a drive measured at 398 MB/s. A longer prefill could incur more disk reads if the page cache comes under pressure.
A separate A/B test found no disk effect for a model resident entirely in card memory, but that test does not resolve the host-resident case. The storage explanation remains untested.
The constant-throughput assumption needs revision
The initial cost model assumed Metal prefill stayed at 660 tok/s, based on a measurement near 24k tokens. Both runs show lower throughput near 48k. A constant rate therefore understates the measured long-context benefit.
An earlier prediction put speedup at 4.2× at 24k and 3.2× at 192k, assuming KV-cache growth would displace resident experts. The tested range shows the opposite direction. It does not establish behaviour at 192k, and this all-experts-on-host configuration does not directly test expert displacement.
The cost of a second microbatch
llama.cpp processes long prompts in microbatches, called ubatches. In this configuration, each holds up to 24,576 tokens and requires another transfer of host-resident experts. The original six-point fit covered only single-microbatch prompts, so it could not measure the repeated transfer cost.
A 23,691-token prompt takes 15.0 seconds; a 26,156-token prompt takes 22.0 seconds. Each was repeated twice with matching timings. The fitted compute rate accounts for approximately 0.8 seconds of the increase; the remaining 6.2 seconds is consistent with another expert transfer.
Scroll the table to view all columns
| Prompt | Microbatches | Per-prompt model | Per-microbatch model | Measured |
|---|---|---|---|---|
| 23,691 tokens | 1 | 15.0 s | 13.5 s | 15.0 s |
| 26,156 tokens | 2 | 16.2 s | 22.0 s | 22.0 s |
An earlier test used four evenly spaced prompts that could fit either a step or a smooth slope. Sampling immediately on both sides of the boundary distinguished those explanations.
Revised RTX 5090 cost model
Four prefill measurements fit the expression below with a root-mean-square residual of 0.10 seconds. The transfer term is paid once per microbatch.
time = ⌈tokens ÷ 24,576⌉ × 6.21 s + tokens ÷ 3,300 + 1.65 sPreviously estimated at 8.00 s
Previously estimated at 5.46 GiB/s
Previously estimated at 5,150 tok/s
These are fitted parameters for Qwen3-Coder-Next in this configuration, not general hardware limits. The cross-model study explains why the same fitting approach was unsuitable for its sweep, which included cold-start measurements.
Choosing a model: compute and memory
Two properties matter: the computation required per token and the amount of model data that must cross Thunderbolt.
Active parameters
A mixture-of-experts model uses only a subset of its experts for each token. Total parameter count therefore gives an incomplete picture of the work involved. The working hypothesis is that more active parameters increase the benefit of CUDA prefill. The six-model study tests this relationship beyond Qwen3-Coder-Next.
Memory requirements
Weights, the key-value (KV) cache and working buffers all consume memory. A model that fits entirely on the RTX 5090 can use the direct CUDA path. Split inference targets larger MoE models whose experts can stream during prefill and whose generation stage fits in the Mac’s available memory. File size alone does not establish either fit.
Expert reuse differs between prefill and generation. A cache policy that helps token-by-token generation may be less effective for a long prompt. This configuration transfers every expert per microbatch and does not assume a particular routing distribution.
Screening a model before downloading it
tools/disagg-screen.py reads the GGUF header to estimate transfer volume, relative prefill
time and a crossover prompt length. It reads only a few megabytes, without mapping model weights or opening the
card. Dense models larger than the GPU are excluded because this workflow depends on expert streaming.
Corrections to the screening estimates
Two issues were corrected on 20 September. Rankings produced before that date may overstate the benefit:
- Sharded model files. The tool previously counted only the first shard. One 69 GB model was reported as 37 GB, with 7.3 GiB of expert transfer and a 5.5× estimated gain. The corrected figures were 69 GB, 39 GiB of transfer and a 3.4× estimated gain.
- Compute-buffer reservations. The tool previously assigned the full 30 GiB budget to resident experts, omitting the 8–11 GiB compute buffer reserved for a 24,576-token microbatch. The resident-expert budget is now 21 GiB.
The tool still assumes constant Metal throughput. Its long-context projections remain provisional until that part of the model is revised.
What this is for
Some work cannot leave the building. Court transcripts, patient records, case files, documents under legal hold, an unreleased codebase — the material that most needs careful reading is often exactly the material you are not permitted to send to an API. For that work the choice is not cloud or local. It is local or not at all.
That constraint has a second edge. When accuracy is what matters, you want the largest model you can run, not the fastest one — and the models worth running for that are 50–100 GB, which an Apple Silicon Mac can hold but cannot process quickly. This is the intersection this path exists for: work that must stay on the machine, against a model chosen for accuracy, on hardware that makes it impractical.
Mixtral 8x22B processing a 48,000-token prompt on the Mac's GPU takes 615 seconds — ten and a quarter minutes before the first token of the answer. The same prompt on the RTX 5090 takes 59 seconds. Nobody waits out the first figure document after document; that is the difference between a workflow and a demonstration.
Measured on a real task
Eighteen source files from the VLC codebase, audited for a genuine question — does this file write outside the application folder — with the same model, the same prompts and the same token limits on both halves. Long input, short structured answer, nothing cached between files.
| Input | Metal only | Split | Gain |
|---|---|---|---|
| 23,278 tokens | 33.8 s | 12.6 s | 2.67× |
| 19,461 tokens | 26.9 s | 9.5 s | 2.83× |
| 11,367 tokens | 14.8 s | 7.3 s | 2.03× |
| all eighteen files | 251 s | 146 s | 1.72× |
Qwen3-Coder-Next 80B-A3B, 20 September 2026. End-to-end wall time per request, including the
state transfer between the two GPUs — not prefill in isolation. Both halves produced identical findings on all
eighteen files, including the real one: VLC's media library writes ml.db into the user data
directory.
Extending the same arithmetic to a 67,000-token document — the median book in a 1,401-volume scanned corpus — gives roughly 3.4× for extraction or flagging, where the answer is short, and 2.5× for a written summary, where a couple of thousand generated tokens dilute it. Generation runs on the same GPU in both arrangements, so the advantage survives only while the input dominates the output; it disappears somewhere past 7,000 generated tokens. No legal or medical corpus has been run through this path. The document figures are arithmetic from measured rates, and the VLC row above is the only end-to-end measurement of a real task.
One practical note that cost a night to learn: the state handed between the two GPUs is hundreds of megabytes per request, and where it is written is on the critical path. Writing it to a full, slow disk reduced the gain above from 1.72× to 1.03× — the entire benefit, erased by a default path.