How it works
Throughput
How fast llamay runs depends on the model, its quantisation, the hardware and the settings far more than on anything a page can promise. So this page is about measuring it on your own machine: which command, what each number means, and which settings move them.
Measure it
llamay bench -m qwen2.5:0.5b -prompt 512 -gen 128llamay 0.3.0 · Qwen2.5 0.5B Instruct · qwen2 · 7 threads · kernels neon+i8mm · matmul cpu · gemm own kernels · kv f32
weights 373.71 MiB · repack on · context 32768 · prompt 512 · generate 128
phase rate best note
prefill 169.0 tok/s 3.03s batched, 256 at a time
decode 27.2 tok/s 4.706s one token per pass
ttft 3.03s time to first token
fork x8 7µs 0 pages allocated (a copy would be 64)
snapshot 22ms 12.00 MiB for 512 tokens, at f32
restore 15ms vs 3.03s to re-prefill
Restoring a 512-token context is 199.3x faster than recomputing it.
The darwin_arm64 build of 0.3.0 on an Apple M4, CPU only, best
of three runs.
The same command with the darwin_arm64_metal build and
-gpu, on the same machine and file:
llamay 0.3.0 · Qwen2.5 0.5B Instruct · qwen2 · 7 threads · kernels neon+i8mm · matmul metal:Apple M4 · gemm accelerate · kv f32
prefill 1200.5 tok/s 426ms one command buffer per 256-token tile
decode 54.7 tok/s 2.34s 506 dispatches per token, one submissionSeven times the prefill and twice the decode, from the build alone. That is the first thing to check when a number looks low.
| Line | Means | Bound by |
|---|---|---|
prefill | How fast the prompt is read. | Arithmetic: many tokens per pass, so the multiplies dominate. |
decode | How fast the answer is written, one token per pass. | Memory bandwidth: every token reads all the weights once. |
ttft | Time to the first token of the answer: the prefill, for this prompt. | Prompt length. |
fork, snapshot, restore | What the state engine costs. | Page tables, and the disk. |
The same command, recorded:
Transcript — the whole session as text
~/llamay $ llamay bench -m qwen2.5:0.5b -gpu -prompt 512 -gen 128 -reps 3 llamay v0.1.58-1-gb14f550 · Qwen2.5 0.5B Instruct · qwen2 · 7 threads · kernels neon+i8mm · matmul metal:Apple M4 · gemm own kernels · kv f32 weights 373.71 MiB · repack on · context 32768 · prompt 512 · generate 128 phase rate best note prefill 794.3 tok/s 645ms one command buffer per 256-token tile decode 47.3 tok/s 2.705s 506 dispatches per token, one submission cpu encoding 133ms · gpu execution 3.175s (4% encoding) phase rate best note fork x8 11µs 0 pages allocated (a copy would be 64) snapshot 35ms 12.00 MiB for 512 tokens, at f32 restore 34ms vs 645ms to re-prefill gpu time: staging 41ms · kernels 2.333s · copy-back 50ms Restoring a 512-token context is 18.9x faster than recomputing it. ~/llamay $ # prefill is arithmetic-bound; decode is memory-bound. One number would hide that .
Other measurements
llamay run -statsprints what one real generation cost, including what a JSON Schema or grammar mask added.llamay benchforkmeasures many questions asked of one shared prefix, forked against a warm prefix tree, which is where contexts pay.llamay bench -sweepmeasures prefill across tile widths;-spec-*flags compare speculative decoding against plain decoding on the same text.- On a server,
/metricsand traces split each request into queue, prefill and decode. See observability.
What the state engine is worth at a prompt size worth keeping, with thirty-two branches off it:
Transcript — the whole session as text
~/llamay $ llamay bench -m qwen2.5:0.5b -gpu -forks 32 -prompt 2048 -gen 32 -reps 1 llamay v0.1.58-1-gb14f550 · Qwen2.5 0.5B Instruct · qwen2 · 7 threads · kernels neon+i8mm · matmul metal:Apple M4 · gemm own kernels · kv f32 weights 373.71 MiB · repack on · context 32768 · prompt 2048 · generate 32 phase rate best note prefill 650.4 tok/s 3.149s one command buffer per 256-token tile decode 41.7 tok/s 768ms 506 dispatches per token, one submission cpu encoding 40ms · gpu execution 3.854s (1% encoding) phase rate best note fork x32 65µs 0 pages allocated (a copy would be 1024) snapshot 129ms 48.01 MiB for 2048 tokens, at f32 restore 74ms vs 3.149s to re-prefill gpu time: staging 149ms · kernels 9.77s · copy-back 191ms Restoring a 2048-token context is 42.6x faster than recomputing it. ~/llamay $ # 0 pages allocated where a copy would have been 1024 — that is the fork
What moves the numbers
| Setting | Effect |
|---|---|
| The build | The plain macOS tarball has no Metal; _metal does. On Linux, _gpu carries CUDA and Vulkan. llamay version says what is built in; /healthz says whether the GPU is in use. |
| Quantisation | Decode reads every weight per token, so a q4_k_m file decodes faster than a q8_0 of the same model. Check what it cost with llamay ppl. |
-kv vq8 | A smaller cache: more context and more concurrent requests in the same memory. The trade. |
-batch N on serve | Several requests decoded together read the weights once, so total throughput rises with concurrency. Batch composition never changes the output. |
-concurrency, -prefill-chunk | How many requests run at once, and how much of a long prompt is read per step so it does not stall the others. |
| The prefix cache and contexts | A prompt whose start is already computed is not computed again. Keep stable text at the start, and see cached_tokens. |
-spec ngram, -lookup, -md | Speculative decoding: draft tokens from the prompt, or from a smaller model, and verify them in one pass. Pays when the answer repeats the input — editing, extraction — and costs a little when it does not. |
-t | Worker threads. The default avoids efficiency cores, which would otherwise hold the rest back. |
Free throughput where a prompt repeats itself, which is what editing looks like:
Transcript — the whole session as text
~/llamay $ # a prompt that repeats itself, which is what an edited document looks like
~/llamay $ wc -c build/demo/work/repeat.txt
1464 build/demo/work/repeat.txt
~/llamay $ # first the case it is for: the answer quotes the prompt back
~/llamay $ llamay run -m qwen2.5:0.5b -n 96 -stats -p "$(cat build/demo/work/repeat.txt)
> Repeat the paragraph above, word for word:"
The quarterly report states that revenue grew by 14 percent. The quarterly report states th
at revenue grew by 14 percent. The quarterly report states that revenue grew by 14 percent.
The quarterly report states that revenue grew by 14 percent. The quarterly report states tha
t revenue grew by 14 percent. The quarterly report states that revenue grew by 14 percent. T
he quarterly report states that revenue grew by 14 percent. The quarterly report states that
prompt 322 tokens in 2.323s (138.6 tok/s) · generated 96 in 7.991s (12.0 tok/s) · stop: leng
th
~/llamay $ llamay run -m qwen2.5:0.5b -lookup -n 96 -p "$(cat build/demo/work/repeat.txt)
> Repeat the paragraph above, word for word:"
The quarterly report states that revenue grew by 14 percent. The quarterly report states th
at revenue grew by 14 percent. The quarterly report states that revenue grew by 14 percent.
The quarterly report states that revenue grew by 14 percent. The quarterly report states tha
t revenue grew by 14 percent. The quarterly report states that revenue grew by 14 percent. T
he quarterly report states that revenue grew by 14 percent. The quarterly report states that
prompt 322 tokens in 2.102s (153.2 tok/s) · generated 96 in 5.083s (18.9 tok/s)
draft ngram · 96 tokens in 15 passes (6.40x) · 81/88 drafts accepted (92%) · 6.40 tokens per
target pass · lookahead 8
~/llamay $ # now the case it is not for: a summary shares almost nothing with the prompt
~/llamay $ llamay run -m qwen2.5:0.5b -lookup -n 96 -p "$(cat build/demo/work/repeat.txt)
> Summarise:"
What is the answer to the question where is the word "growth" in the context of the sentenc
e? The sentence is about a company's growth in revenue. The word "growth" in the context of
the sentence is used to describe the company's increase in revenue over a certain period. Sp
ecifically, the sentence states that the company's revenue grew by 14 percent. The word "gro
wth" is used in this context to indicate that the company's revenue has increased in value
prompt 317 tokens in 2.302s (137.7 tok/s) · generated 96 in 8.052s (11.9 tok/s)
draft ngram · 96 tokens in 79 passes (1.22x) · 17/38 drafts accepted (45%) · 1.22 tokens per
target pass · lookahead 8
~/llamay $ # the acceptance rate in the last line is the whole story — it is what decides wh
ether drafting is free or wasted
Paths that are slower on purpose
- Images are read on the CPU path, a minute or two per image for a 4B model on a laptop.
- Logprobs and image requests skip the batching scheduler: they run one at a time.
- LoRA adapter requests run on the per-matmul path rather than a whole-model GPU graph.
- Constrained output costs a mask per token, typically tens of per cent, reported by
run -stats. - The i-quants and MXFP4 run on portable Go code on the CPU, not assembly or a GPU kernel. Coverage
Measuring fairly
- Use a real prompt length. A 30-token prompt measures request overhead, not prefill.
- Vary the prompt between runs when measuring a server. A repeated prompt is answered from the prefix cache and looks impossibly fast.
- Take the best of several runs, as
benchdoes, and report the machine, the build and the file with the number. - Measure quality too. A faster quantisation that is worse is not faster at the job. Correctness has the checks.