How it works
Models and formats
Seventeen architectures reached by thirty-nine declared names, twenty-three tensor formats, three tokenizer families and three KV cache precisions. Anything outside that is refused at load with a message naming what does work — never loaded partially and run with the missing piece skipped.
Architectures
| Declared | Covers | What makes it different |
|---|---|---|
llama mistral azmx | Llama 1–3, Mistral, AZMX One and Code, most fine-tunes | The baseline: GQA, interleaved rotary, RMSNorm, SwiGLU. |
qwen2 qwen3 | Qwen 2, 2.5 and 3 | The neox rotary convention; qwen3 adds per-head query and key norms. |
gemma gemma2 gemma3 | Gemma 1, 2 and 3 | Scaled embeddings, GELU gate, post-attention and post-feed-forward norms, logit softcapping, sliding window on 1-in-2 or 5-in-6 layers with a second rotary base on the local ones. |
phi3 | Phi-3 and Phi-3.5 | One fused attention projection, and a feed forward with the gate stacked on the up projection, split at load. |
stablelm | StableLM 2 | Layer normalisation with biases, and only a quarter of each head rotated. |
gpt2 | GPT-2 and models built on its block | Layer normalisation, learned absolute positions, an ungated feed forward. |
falcon gptneox phi2 | Falcon, Pythia, the GPT-NeoX lineage, Phi-2 | The parallel residual: attention and the feed forward both read the block's input, and both land on the residual. |
qwen3moe qwen2moe, and llama with expert_count | Qwen3-MoE, Qwen1.5-MoE, Mixtral | A router picks k of n per token, experts stored as stacked 3-D tensors, a shared expert where the architecture has one. |
gpt-oss | gpt-oss 20B and 120B | A per-head attention sink in the softmax denominator, MXFP4 experts, YaRN scaling, a clipped SwiGLU gate whose linear half is offset by one, a bias on every expert projection, the o200k pre-tokenizer. |
bert | all-MiniLM and the sentence transformers; the ms-marco cross-encoders | Not a decoder at all: bidirectional attention, post-norm blocks, WordPiece, mean pooling. With a classification head and a pair input, a reranker. |
azmx-ocr | The AZMX recogniser | A CTC head over a script alphabet. |
Reading a file before trusting it:
Transcript — the whole session as text
~/llamay $ llamay info -m qwen2.5:0.5b llamay v0.1.58-1-gb14f550 kernels neon+i8mm gpu backend metal:Apple M4 file ~/.llamay/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc8 70eda801d3ab0515 gguf version 3 size 379.38 MiB tensors 290 architecture qwen2 (decoder) name Qwen2.5 0.5B Instruct layers 24 embedding 896 heads 14 query / 2 kv (group 7) head dim 64 feed forward 4864 context 32768 rope base 1e+06 kv per token 24.00 KiB tokenizer gpt2, 151936 tokens, pre qwen2 bos / eos 151643 / 151645 prepends bos false quantisation Q5_0 x132, F32 x121, Q8_0 x13, Q6_K x12, Q4_K x12 tensor audit 290 of 290 tensors read, none unread ~/llamay $ # the name on the file is a claim; the tensor histogram is the fact
Tensor formats
Twenty-three read: F32, F16, BF16, Q4_0/1, Q5_0/1, Q8_0, Q8_1, Q2_K through
Q8_K, MXFP4, and the i-quants IQ4_NL, IQ4_XS, IQ2_XXS, IQ2_XS, IQ2_S,
IQ3_XXS, IQ3_S. llamay info -m file.gguf -tensors reports what
one file is, including whether any tensor in it went unread.
| Family | Read | Written by llamay quantize | Accelerated |
|---|---|---|---|
| F32, F16, BF16 | yes | yes | every backend |
| Q4_0/1, Q5_0/1, Q8_0 and the K-quants | yes | yes | assembly on both CPU architectures, and every device backend |
| MXFP4 | yes | fixtures only | portable Go only — the codebook needs a byte shuffle, not a shift |
| IQ2, IQ3, IQ4 | yes | no | portable Go only — a codebook lookup is a gather, which is what NEON and AVX2 are worst at |
| IQ1_S, IQ1_M | refused outright | no | — |
Picking grid points well is a search against an importance matrix, and a
bad i-quantiser is worse than none — it produces a file that loads, runs,
and is quietly worse than the one it replaced. llamay imatrix
exists to measure which columns a model drives; the quantiser that would
use that measurement does not.
And the same file at full depth — every tensor, every metadata key, and a tokenizer round trip on your own text:
Transcript — the whole session as text
~/llamay $ llamay info -m qwen2.5:0.5b -tensors | head -20 llamay v0.1.58-1-gb14f550 kernels neon+i8mm gpu backend metal:Apple M4 file ~/.llamay/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc8 70eda801d3ab0515 gguf version 3 size 379.38 MiB tensors 290 architecture qwen2 (decoder) name Qwen2.5 0.5B Instruct layers 24 embedding 896 heads 14 query / 2 kv (group 7) head dim 64 feed forward 4864 context 32768 rope base 1e+06 kv per token 24.00 KiB tokenizer gpt2, 151936 tokens, pre qwen2 bos / eos 151643 / 151645 prepends bos false ~/llamay $ llamay info -m qwen2.5:0.5b -meta | head -18 llamay v0.1.58-1-gb14f550 kernels neon+i8mm gpu backend metal:Apple M4 file ~/.llamay/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc8 70eda801d3ab0515 gguf version 3 size 379.38 MiB tensors 290 architecture qwen2 (decoder) name Qwen2.5 0.5B Instruct layers 24 embedding 896 heads 14 query / 2 kv (group 7) head dim 64 feed forward 4864 context 32768 rope base 1e+06 kv per token 24.00 KiB tokenizer gpt2, 151936 tokens, pre qwen2 ~/llamay $ llamay info -m qwen2.5:0.5b -tokenize README.md llamay v0.1.58-1-gb14f550 kernels neon+i8mm gpu backend metal:Apple M4 file ~/.llamay/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc8 70eda801d3ab0515 gguf version 3 size 379.38 MiB tensors 290 architecture qwen2 (decoder) name Qwen2.5 0.5B Instruct layers 24 embedding 896 heads 14 query / 2 kv (group 7) head dim 64 feed forward 4864 context 32768 rope base 1e+06 kv per token 24.00 KiB tokenizer gpt2, 151936 tokens, pre qwen2 bos / eos 151643 / 151645 prepends bos false quantisation Q5_0 x132, F32 x121, Q8_0 x13, Q6_K x12, Q4_K x12 tensor audit 290 of 290 tensors read, none unread tokenize README.md bytes 62197 tokens 16566 bytes per token 3.754 round trip exact — decode(encode(text)) is the input, byte for byte per line 522 of 522 non-empty lines exact
The KV cache, in three precisions
The format that matters for memory at run time is the cache, and it is offered in three precisions rather than one. The split between them is a measurement rather than a preference.
| Mode | Cost | What it quantises | When |
|---|---|---|---|
f32 | 100% | Nothing. Exact. | The default, and what to use when a model's verify row is red. |
vq8 | 66% | Values. Keys stay exact. | The recommended trade on everything that clears the bound. |
q8 | 31% | Both. | Offered as a note, not a recommendation. |
On a real prompt, quantising keys alone shifts final logits by 11.9% of their
range and quantising values alone shifts them 1.8% — because a key feeds a dot
product that a softmax then exponentiates. That asymmetry is the whole reason
vq8 exists as a separate mode.
gpt-oss moves 12.7% of the sampler's mass under vq8
and phi3-mini moves 5.6%, both over the 5% bound. Run those with
-kv f32. llamay verify reports it per file, so this
is a thing you are told rather than a thing you notice.
The bound, and why it is the one it is
The check used to fail two real files, and the check was wrong rather than the files. It asked whether the largest logit difference anywhere in the vocabulary was a small fraction of the peak-to-peak logit range. Both halves are extreme-value statistics over the whole vocabulary and neither is about the top of the distribution: the numerator landed at ranks 4,671 to 226,526 on real files — nowhere near a token any model would emit — and the denominator is the distance down to an untrained embedding row.
What it asserts now is the mean total-variation distance between the exact-cache and quantised-cache distributions over every position, bounded at 5%: the probability a sampler notices. It is the mean and not the worst position, because a mixture reroutes an expert on a 1e-3 error and worst-position cannot separate a healthy cache from a destroyed one at all.
| Cache bits | Measured TV distance across 14 fixtures and 7 real files |
|---|---|
| 8 | 0.17 – 4.6% |
| 4 | 3.0 – 10.1% |
| 2 | 7.6 – 33.4% |
A permanent vq8 bound has teeth row runs the same comparison against a one-bit cache and fails if that passes, so the row above it can never be vacuous.
Tokenizers
Byte-level BPE with the GPT-2, Qwen 2, Llama 3 and o200k pre-tokenizers —
hand-written, since RE2 has no lookahead — plus SentencePiece and WordPiece.
A file whose tokenizer.ggml.pre names a rule set llamay does not
implement still loads, under a guess that Tokenizer.PreKnown
records rather than hides.
AZMX's vocabulary beats every comparable tokenizer on Indic fertility by
2.1× to 6.5×. An engine that reproduces its input only approximately gives
that advantage back, and does so silently — the text comes out looking
almost right. Two real architectures failed exactly this: a byte-level
vocabulary is not uniformly byte-level, Pythia's carries 23 entries of
literal spaces marked USER_DEFINED, and putting them through
the byte decoder drops a real space.
What a quantised cache costs, at the only place the question can be settled — the text:
Transcript — the whole session as text
~/llamay $ llamay run -m qwen2.5:0.5b -kv f32 -seed 3 -n 64 -p "The reason a KV cache domina tes memory at long context is" ____ A. to improve the efficiency of accessing memory B. to improve the efficiency of accessing data C. to improve the efficiency of accessing the CPU D. to improve the efficiency of accessing the GPU Answer: A Which of the following statements about the application of the VCR (Video Cassette prompt 11 tokens in 112ms (98.0 tok/s) · generated 64 in 5.093s (12.6 tok/s) · stop: length ~/llamay $ llamay run -m qwen2.5:0.5b -kv vq8 -seed 3 -n 64 -p "The reason a KV cache domina tes memory at long context is" ________. A. It can be used as a write-through cache B. It can be used as a read-only memory C. It can be used as a volatile memory D. It can be used as a non-volatile memory Answer: C In the process of educational activities, the most prompt 11 tokens in 119ms (92.7 tok/s) · generated 64 in 5.216s (12.3 tok/s) · stop: length ~/llamay $ # the bound on that difference is measured, not asserted — see `llamay verify`
Where models come from
| Source form | Meaning |
|---|---|
<name>[:<tag>] | An Ollama-style registry, registry.ollama.ai by default. |
hf.co/<owner>/<repo>[:<quant>] | A Hugging Face repository; the tag picks the quantisation. |
hf:<owner>/<repo>/<file.gguf>[@rev] | An exact file, optionally pinned to a commit. |
https://… | Any URL serving a GGUF — an internal mirror, an object store, a colleague's machine. |
| A path | A file that exists always wins over a name. No store on the machine can change what a path means. |
llamay fetches from registries and hosts nothing. There is no llamay registry
and there will not be one: hosting weights is a bandwidth business with
nothing defensible in it, and an engine that requires its own registry has
made itself the gatekeeper of its own models. That is why
llamay pull exists and POST /api/push is refused by
name.
Requantising a file, and measuring rather than assuming what it cost:
Transcript — the whole session as text
~/llamay $ llamay info -m testdata/azmx-tiny.gguf | head -8
llamay v0.1.58-1-gb14f550
kernels neon+i8mm
gpu backend metal:Apple M4
file testdata/azmx-tiny.gguf
gguf version 3
size 2.39 MiB
tensors 38
architecture azmx (decoder)
~/llamay $ llamay quantize -i testdata/azmx-tiny.gguf -o build/demo/work/tiny-q4.gguf -type
q4_k
29 tensors converted, 9 left as they were
25 of them are Q4_0 rather than Q4_K: their rows are not a multiple of 256
2.38 MiB -> 346.64 KiB (14.2% of the original) in 12ms
~/llamay $ ls -lh testdata/azmx-tiny.gguf build/demo/work/tiny-q4.gguf | awk '{print $5, $9}
'
353K build/demo/work/tiny-q4.gguf
2.4M testdata/azmx-tiny.gguf
~/llamay $ llamay info -m build/demo/work/tiny-q4.gguf | head -8
llamay v0.1.58-1-gb14f550
kernels neon+i8mm
gpu backend metal:Apple M4
file build/demo/work/tiny-q4.gguf
gguf version 3
size 353.25 KiB
tensors 38
architecture azmx (decoder)
An importance matrix first, so the scale search knows which columns your corpus actually drives:
Transcript — the whole session as text
~/llamay $ head -c 4000 README.md > build/demo/work/calib.txt && wc -w build/demo/work/calib
.txt
620 build/demo/work/calib.txt
~/llamay $ llamay imatrix -m testdata/azmx-tiny.gguf -f build/demo/work/calib.txt -chunks 4
-o build/demo/work/imatrix.dat
4 chunks, 2048 tokens
28 matrices, 2048 activations each
column spread: 3.3x typical, 8.4x worst (blk.3.ffn_down.weight)
a spread near 1 means every column matters alike and calibration will not help;
the further above 1, the more a weighted scale search has to work with
wrote build/demo/work/imatrix.dat
llamay quantize -i testdata/azmx-tiny.gguf -o out.gguf -type q4_k -imatrix build/demo/work/i
matrix.dat
~/llamay $ llamay quantize -i testdata/azmx-tiny.gguf -o build/demo/work/tiny-q4-im.gguf -ty
pe q4_k -imatrix build/demo/work/imatrix.dat
29 tensors converted, 9 left as they were
25 of them are Q4_0 rather than Q4_K: their rows are not a multiple of 256
2.38 MiB -> 346.64 KiB (14.2% of the original) in 43ms
What is refused
- Mamba and the state-space models — declined at load rather than approximated.
- Vision encoders — CPU path only (
-mmproj); no GPU backend reads images yet. - DeepSeek-V2 and V3 — refused by name, and the reason is not a missing router: multi-head latent attention caches one shared compressed latent per position instead of a key and a value per head, so it changes what a KV page holds. That is the paged allocator, the copy-on-write fork and the snapshot format, not the attention alone.
- The OCR encoder — the decoder is here, because on that model the decoder is where the accuracy is. Frames come in from the recogniser; the encoder graph follows. A scan with no recogniser behind it is read by a vision model instead:
llamay ocr -image. - LoRA on a device graph — adapters are applied at the multiply on the per-matmul path; a device graph runs whole blocks and applies none, so an adapter's requests do not use one.