CUDA inference on a Mac.
Powered by an RTX 5090.
macuda connects an Apple Silicon Mac to an NVIDIA GPU over Thunderbolt. A custom driver and CUDA compatibility layer run the unmodified CUDA backends of llama.cpp and stable-diffusion.cpp, with dense-model decode throughput at 97–98% of the Windows reference in the published benchmark batch.
A CUDA stack built for macOS
Running an NVIDIA GPU from an Apple Silicon Mac requires more than a Thunderbolt connection. The applications need a CUDA runtime, a GPU driver and a way to access the hardware. macuda provides those layers for the workloads tested here.
The project combines three C libraries with a split compilation workflow. Together, they initialise the GPU, manage memory and execute CUDA kernels from native macOS applications. The host application is compiled for the Mac; the GPU code is compiled with NVIDIA's toolchain on Linux.
The engineering approach. Keep the applications' CUDA backends intact and implement the interfaces beneath them, from memory allocation and kernel launches to firmware initialisation and hardware queues.
From application to GPU
The application calls a CUDA compatibility layer, which submits work through the userspace driver. tinygrad's DriverKit extension provides hardware access over Thunderbolt.
The driver layer is a C port of the userspace NVIDIA driver in
tinygrad, using NVIDIA's own published structure definitions from open-gpu-kernel-modules. The
GPU runs NVIDIA's signed GSP firmware, downloaded from linux-firmware and hash-checked. NVIDIA's headers, firmware
and Linux compiler remain part of the workflow; no NVIDIA driver or runtime binary executes on the Mac.
One application, two compilation targets
A CUDA application contains host code for the CPU and device code for the GPU. macuda compiles those components separately, then combines them into a native macOS executable.
- GPU kernels: Linux
nvccruns in a Docker container on the Mac or on a separate machine over SSH and produces machine code for the RTX 5090. - Host application: Homebrew clang compiles the CPU code on the Mac in CUDA host-only mode and embeds the compiled GPU kernels in the resulting Mach-O objects.
The executable links against macuda's driver and compatibility libraries. Building device code requires the Linux toolchain, locally in a container or over SSH; running an application built with a precompiled kernel archive does not.
Validated workloads
Six language models, up to 35B
Dense, hybrid and mixture-of-experts models reach 91–101% of Windows reference decode throughput. Prompt-processing results vary by model and are reported separately.
An OpenAI-compatible API
llama-server with speculative decoding reached 124 tok/s on a first single-stream request and
approximately 207 tok/s across eight concurrent slots. A 20-minute test completed 440 requests with no errors.
Three diffusion models
SDXL Turbo in 2.78 s, Stable Diffusion 1.5 in 3.71 s and Z-Image Turbo at 1024 × 1024 in 10.06 s. PNG outputs matched the project's validation runs byte for byte.
GPU prefill, Metal decode
A 49.6 GB model can use the RTX 5090 for prompt processing, then transfer its state to the Mac for generation. A 23,692-token prompt took 15.2 s on the RTX 5090 versus 35.7 s on Metal in the 16 September test.
Scope and limitations
- Performance depends on the workload. Qwen3.5 35B-A3B reaches 91% of reference decode throughput in the published batch. The other five models reach 97–101%. Profiling identifies host submission overhead as a remaining constraint.
- Hardware access requires coordination. The Thunderbolt setup uses a single-process lock and preflight checks. Some firmware or bus faults require physically reconnecting the enclosure.
- This is a research implementation. The compatibility libraries cover the CUDA interfaces used by the tested applications. PyTorch and vLLM support has not been established.
- Results include their validation context. Operator checks and output comparisons establish correctness. Performance changes are evaluated with interleaved A/B runs on the same host; the Windows reference was measured separately.
Explore the project
Benchmarks
Decode, prompt processing, speculative decoding and image generation, with hardware details, validation criteria and a Windows reference using the same physical GPU.
Deep diveArchitecture
Firmware initialisation, CUDA compatibility, kernel submission and the profiling that guides performance work. Includes the development method and documented corrections.
SourceRepository
Driver and compatibility libraries, build instructions, operating tools and the design records behind the published results.