Concepts

Backends and hardware

One binary covers every CPU Go targets. Each accelerator is a build tag, because each is a cgo shim and the default build has no cgo at all — which is the property that makes make cross six targets from one machine.

The builds

Build	Tag	Needs to build	Needs to run
`make build`	—	Go 1.26	Nothing. NEON + DotProd and i8mm on ARM; AVX2 and AVX-512 VNNI at 256 and 512 bits on x86.
`make build-metal`	`metal`	macOS and Xcode tools	Apple Silicon.
`make build-fast`	`metal accelerate`	macOS	Apple Silicon. Adds Apple's BLAS and the AMX unit.
`make build-cuda`	`cuda`	A C compiler. No toolkit, no headers, no card.	An NVIDIA driver, compute capability 6.1 or later.
`make build-vulkan`	`vulkan`	A C compiler. No SDK, no loader. The ABI is transcribed.	A Vulkan loader and driver.
`make build-gpu`	`cuda vulkan`	A C compiler	Either driver; the backend is chosen at run time.

dlopen, not link

Both discrete backends resolve their driver at first use rather than linking against it, so one Linux binary runs on a machine with a card and on one without, declining cleanly instead of failing to start. CI asserts this with ldd on every build, because it is the kind of property that regresses the moment somebody adds a convenient import.

How the binary decides

Three tiers, and the fall from one to the next is per weight format rather than per model. That is what stops one unsupported tensor from moving an entire forward pass back to the CPU.

CPU

Hand-written assembly for all eight block formats on both architectures, and a portable Go reference everywhere else which is correct and 2.8× to 7.6× slower per dot product on an M4. The hot shape is a W4A8 fused dot product: the activation is quantised to int8 once per matmul and weight rows are never materialised.

The CPU is not a fallback that exists on paper. make cross-portable builds riscv64, s390x, ppc64le and wasm on every push, and LLAMAY_BACKEND=portable runs the whole suite through the reference kernels on a machine that has SIMD — because building them is not the same as running them.

Metal

The whole forward pass on the device, and the widest block coverage of the three: thirteen of the fourteen generated shapes run, including all three mixtures, all three parallel-residual blocks and the attention sink. gpt-oss is the one it refuses, twice over — metal.Unsupported declines its expert biases, and NewGraph declines its clipped SwiGLU gate before it gets that far.

TestEveryShapeIsRunOrRefused ranges over the generated shapes rather than over that function's clauses, so a block feature added to the CPU and not implemented on the device fails the build until somebody either implements it or declares it.

Accelerate is not bit-identical

It dequantises weights to float32 for sgemm; the default path keeps them quantised and quantises the activation to int8. Two methods, not one method and its rounding error — 0.09% of perplexity end to end. LLAMAY_NO_BLAS=1 makes an accelerate build exact again, which is the flag to reach for before blaming a model.

A refusal that was costing twelve times the prefill

Gemma was refused on Metal over a device-versus-CPU perplexity difference reported as about one per cent. Re-measured, it is 0.05% — 4.3965 on the device against a CPU reference of 4.3988, and on the better side. The refusal was costing gemma3-270m 47 tok/s on the per-matmul fallback against 552 on the graph.

CUDA

Pascal (2016) onward, seven weight formats, __dp4a for the W4A8 inner product, split-position attention for decode, and a tensor-core prefill kernel on Turing and later. The toolkit is needed only to regenerate the PTX, which is checked in.

Below compute capability 6.1 it declines at open with a message naming the reason, rather than running through a slower kernel that would need separate testing. On a Tesla T4 it reaches 38% of llama.cpp's prefill and 52% of its decode; after the attention rewrite the tensor-core matmul is 47% of a prefill, and the remaining gap is the one llama.cpp's MMQ has and this kernel does not — staging the next weight tile behind the current multiply.

Repacking is a per-device decision now

llamay used to rewrite Q5_0 weights to Q8_0 at load. That is worth 2.9× on a CPU, cost 10% of decode on a T4, and helps on Metal, whose unified memory makes the same trade come out the other way. tensor.Repack asks the device instead — CUDA reads CU_DEVICE_ATTRIBUTE_INTEGRATED, Vulkan reads VkPhysicalDeviceProperties.deviceType, Metal always shares — which on a T4 is worth +10.4% decode at +0.4% prefill.

Vulkan

AMD, Intel, Android, and NVIDIA as a fallback. The whole forward pass on the device, ten quantised weight formats plus F32/F16/BF16, and VK_KHR_shader_integer_dot_product where the driver has it.

It is the widest gap in this engine and is written down that way rather than presented as parity. It runs on real cards and is still several times behind llama.cpp's own Vulkan on the same T4, interleaved. The seven i-quants have no Vulkan kernel and are declined by name at both tiers, because an i-quant decodes through a per-format codebook rather than arithmetically and that is a different kind of kernel.

Cooperative matrix has a kernel — all ten quantised formats, worth +44% of prefill on a T4 — and it is the one kernel here that is not bit-identical to the CPU, bounded at 0.1% of the output range. The reason is the operands rather than the accumulation order: this hardware's cooperative matrix takes float16, and a dequantised W4A8 weight does not fit in float16 exactly. That bound is stated rather than hidden.

What each backend refuses

Block feature	CPU	Metal	CUDA	Vulkan
Dense decoders — Llama, Qwen, Gemma 2/3, Phi-3, GPT-2, StableLM	yes	yes	yes	yes
Mixture of experts — Mixtral, Qwen2-MoE, Qwen3-MoE	yes	yes	declines	declines
Parallel residual — Falcon, GPT-NeoX, Phi-2	yes	yes	declines	declines
Attention sinks	yes	yes	declines	declines
`gpt-oss` — MXFP4 experts, clipped SwiGLU, expert biases	yes	declines	declines	declines
The i-quants — IQ2, IQ3, IQ4	portable Go only	no kernel	no kernel	declined by name
Mamba / SSM, vision encoders, DeepSeek-V2 latent attention	refused at load	—	—	—

A model a device graph declines still runs correctly on that machine: the CPU path is the one that executes, and Model.DeviceUnsupported is the check CUDA and Vulkan make before building a graph. One function answering for three backends can only be as permissive as the least capable, which is why Metal answers metal.Unsupported separately.

Environment

Variable	Effect
`LLAMAY_BACKEND=portable`	Force the reference Go kernels. The way to tell a SIMD bug from a block bug.
`LLAMAY_GPU`	Force a backend on or off.
`LLAMAY_NO_BLAS=1`	Make an `accelerate` build bit-exact again.

llamay version prints what a given build actually carries: its tags, its kernel sets, the GPU backends compiled in, the formats and the architectures. That is the first command to run when a number looks wrong.