The AZMX inference engine
llamay runs GGUF models on your machine from one Go binary, and lets you fork a conversation, write it to disk, and pick it up on another machine. Every other engine recomputes the prompt.
macOS and Linux. On Windows: irm https://llamay.com/install.ps1 | iex.
No account, no card, nothing leaves the machine.
Three commands
Install, pull a model, talk to it. llamay has its own store and its own downloader, so nothing else has to be installed first — and it reads a model Ollama already pulled, as a courtesy rather than a dependency.
serve with no -m takes the first model in your
store and listens on 127.0.0.1:11435 — beside Ollama's
11434, and deliberately away from 8081.
$ llamay pull qwen2.5:0.5b pulling manifest pulling c5396e06af29 [========================] 100.0% 379.4 MiB / 379.4 MiB (26.7 MiB/s) verifying sha256 digest stored qwen2.5:0.5b (379.4 MiB) $ llamay run -m qwen2.5:0.5b -p "The capital of France is" Paris. It has been the capital since 987, when Hugh Capet… $ llamay serve llamay v0.4.0 metal+accelerate 17 architectures listening on http://127.0.0.1:11435
What you get
No CGo and nothing outside the Go standard library on the default build. The GPU backends are one build tag each.
A context is pages, and a fork shares them. Forking a 32-page context eight ways allocates zero pages. Restoring a 512-token context is 91× faster than re-prefilling it.
The state engine →OpenAI, Anthropic and Ollama shapes, in their own conventions — SSE for one, newline-delimited JSON and nanosecond durations for the other. Point an existing client at it by changing a URL.
HTTP reference →Prefill equals decode. A fork continues with a logit difference of zero, not a small one. llamay verify asserts it on any file you own.
Hand-written NEON, i8mm, AVX2 and AVX-512 VNNI kernels. Metal, CUDA and Vulkan resolve their drivers with dlopen, so one Linux build runs with a card and without.
A tool's JSON Schema is compiled to a token-level automaton. The model cannot emit a missing key, a string where a number belongs, or a function nobody offered.
Tools and schemas →A decoder, an encoder and a cross-encoder on one address. Vectors come out unit-length; the reranker agrees with llama.cpp to 3.5e-4 of a logit.
Models and formats →The idea
Inference engines treat a context as memory inside a process. That is right for a chatbot and wrong for an agent, because agents do not run forward in a line — they branch, backtrack and resume.
A radix tree over token sequences returns the longest prefix already computed. Only the new suffix is prefilled — which is what agent traffic is, the same system prompt with one more turn on the end.
A fork copies the page-pointer slice and increments refcounts. A page is cloned only when a branch writes to one whose refcount is above one.
The digest of the weights is recorded and checked on load. KV computed under different weights is not detectably wrong at generation time — it produces confident nonsense — so it fails at load instead.
A reference implementation that shares nothing runs beside it in lockstep, under randomised operation sequences, with the full invariant set asserted after every single step.
Measured
llamay's x86 prefill beats llama.cpp on three of four models. Its decode loses on every platform tested. Those are different problems and the table keeps them apart.
| Model | llamay prefill | llama.cpp | llamay decode | llama.cpp | ||
|---|---|---|---|---|---|---|
| qwen2.5-0.5b | 94.3 | 67.1 | 1.41× | 7.6 | 30.0 | 0.25× |
| llama-3.2-1b | 52.8 | 34.2 | 1.54× | 3.0 | 18.4 | 0.16× |
| gemma-3-1b | 68.1 | 33.7 | 2.02× | 5.5 | 17.2 | 0.32× |
| gpt2-124m | 261.6 | 1060.4 | 0.25× | 24.5 | 213.1 | 0.11× |
It is the one x86 row llamay loses badly, and the one model whose perplexity is 4.67% off llama.cpp's where every other architecture tested is inside 1.3%. Both llamay kernel sets agree with each other exactly on it, which puts the fault in the gpt2 block rather than in a SIMD path. It is the next thing to fix, not a footnote.
Every round of every measurement, the commands that produced them, and the two ways this benchmark lies if taken casually, are in the throughput page.
Drop-in
OpenAI and Anthropic are implemented because those are what everything is written against. Ollama's is implemented in its own shapes — newline-delimited JSON, streaming on by default, durations in nanoseconds — so a client written for it can be pointed here by changing a URL, and then uninstall what it was written for.
The context routes have no counterpart anywhere. They are the point.
Every route, with its body →$ curl -X POST localhost:11435/v1/contexts \ -d '{"prompt":"<the long system prompt>"}' {"id":"ctx_1","tokens":14,"pages":1} $ curl -X POST 'localhost:11435/v1/contexts/ctx_1/fork?n=4' # 4 branches, each holding all 14 tokens, sharing the parent's page $ llamay ctx fork -m model.gguf -i base.llamayctx -n 8 -o branches/ source base.llamayctx, 46 tokens, 1 pages branches 8 in 19µs (2µs each) pages allocated 0 pages a copy would need 8 memory held 128.00 KiB (a copy would be 1.00 MiB)
Before you install
A list of what an engine cannot do is more useful than a list of what it can, and it is the first thing anyone finds out anyway.
On an M4, 68% of llama.cpp's decode rate and 64% of Ollama's. llama-server remains the right answer when raw single-stream throughput is all that matters. Shipping both is a feature.
gpt-oss and phi3-mini exceed the 5% total-variation bound under vq8. Run them with -kv f32; llamay verify says so per file.
State-space models are declined at load rather than approximated. DeepSeek-V2 and V3 are refused by name: latent attention changes what a KV page holds, which is the allocator, the fork and the snapshot format, not the attention alone.
The chat window ships for macOS today. On Windows and Linux llamay is a CLI and a server, which is the whole surface either way.
IQ2, IQ3 and IQ4 load and run on the portable Go kernels only. llamay quantize will not produce one: picking grid points well is a search against an importance matrix, and a bad i-quantiser is worse than none.
Not a limitation, but it belongs in the same list. The server listens on loopback and no provider is configured until you add one. There is no llamay registry and there will not be one.
Free, open source, and everything runs on your machine.