How it works
Correctness and validation
llamay verify compares llamay to llamay. It cannot see a wrongly
wired architecture, because a model rotated the wrong way is rotated the wrong
way consistently — Qwen was rotated the wrong way for the life of this
repository and every check passed throughout. So there are three checks, and
this page is what each catches and what it misses.
Three checks, not one
1 · llamay verify
llamay verify -m qwen2.5:0.5bIt was written before any optimisation. Batched prefill is an optimisation of sequential decode, so the two must agree; a benchmark of an engine where they do not is a benchmark of two different models.
| Row | Asserts |
|---|---|
| prefill ≡ decode | Feeding n tokens as a batch and feeding them one at a time produce the same logits. |
| fork exactness | A forked context continues identically — a logit difference of zero, not a small one. |
| snapshot resume | Writing a context to disk and reading it back changes nothing. |
| tokenizer round trip | Text → ids → text is the input, byte for byte. |
| thread invariance | The worker count does not change the output. |
| batch invariance | Who else is in the batch does not change the output. |
| vq8 bound | Mean total-variation distance between the exact and quantised cache distributions is under 5%. The bound. |
| vq8 bound has teeth | The same comparison against a one-bit cache, which must fail — so the row above it can never be vacuous. |
Eight checks on a tiny test model:
Transcript — the whole session as text
~/llamay $ llamay verify -m testdata/azmx-tiny.gguf check result detail prefill == decode pass max logit difference 0.00e+00 vq8 kv cache pass moves 0.3% of the sampler's mass over 64 positions (worst 0.3% at position 55), greedy agrees at all 64, max logit shift 0.028 — values quantised, key s exact q8 kv cache note moves 0.3% of the sampler's mass over 64 positions (worst 0.4% at position 30), greedy differs at 1 of 64, max logit shift 0.034 — keys quantised too, use vq8 instead vq8 bound has teeth pass a one-bit cache moves 37.2% of the sampler's mass, over th e 5% bound — so the row above is not vacuous on this file tokenizer round trip pass 7 samples exact fork is exact pass difference 0.00e+00 (must be zero) snapshot resumes pass difference 0.00e+00 thread-count invariant pass difference 0.00e+00 across 1, 2 and 4 threads All invariants hold. ~/llamay $ # fork is exact and snapshot resumes: 0.00e+00, which is the only passing value
2 · Greedy text against llama.cpp
Both engines read the same file — llamay pulls it, and
llama.cpp is pointed at the blob llamay's own store wrote — on the same
container, on the CPU, with no GPU anywhere in the harness. Temperature 0,
seed 0, against llama-simple.
llama-simple derives n_ctx from -n,
and gpt-oss's rope scaling is a function of the context it is configured
for. The same prompt through the same reference at -n 32,
-n 88 and -n 424 produces three different
continuations that agree with each other for eight characters. An earlier
run recorded 9 of 34 characters agreeing; at a budget close to llamay's own
window, all 111 agree. Both were correct measurements of two different
reference configurations.
The same eight on weights you would actually serve. The quantised-cache rows move a great deal further:
Transcript — the whole session as text
~/llamay $ llamay verify -m qwen2.5:0.5b check result detail prefill == decode pass max logit difference 0.00e+00 vq8 kv cache pass moves 2.5% of the sampler's mass over 64 positions (worst 6.2% at position 14), greedy differs at 2 of 64, max logit shift 0.617 — values quantised, k eys exact q8 kv cache note moves 16.5% of the sampler's mass over 64 positions (worst 84.4% at position 1), greedy differs at 8 of 64, max logit shift 15.035 — keys quantised to o, use vq8 instead vq8 bound has teeth pass a one-bit cache moves 91.1% of the sampler's mass, over th e 5% bound — so the row above is not vacuous on this file tokenizer round trip pass 7 samples exact fork is exact pass difference 0.00e+00 (must be zero) snapshot resumes pass difference 0.00e+00 thread-count invariant pass difference 0.00e+00 across 1, 2 and 4 threads All invariants hold.
3 · Perplexity
llamay ppl -m model.gguf -f wiki.test.raw -ctx 512| reference | llama.cpp tag b6510, built from source, CPU only |
| runner | 16 vCPU x86-64 (AVX2), Ubuntu 22.04 |
| text | the first 262,144 bytes of wiki.test.raw from wikitext-2-raw-v1, md5 1d92c7cb8f7e4572f40c7e153a818a56 |
| window | 512 tokens, 8 windows, second half of each scored — the scheme llama-perplexity uses, so the two numbers are comparable |
A tenth of a percent of perplexity is two correct implementations disagreeing about float rounding. A whole unit is a bug.
Determinism, demonstrated the only way it can be — the same seed twice, then the same seed on one thread:
Transcript — the whole session as text
~/llamay $ llamay run -m qwen2.5:0.5b -seed 7 -temp 0.7 -n 48 -stats=false -p "A sovereign i nference engine is one that" is able to reason about the world in a way that is consistent with the principles of sovere ignty, such as the principle of non-interference in the affairs of others, and the principle of accountability. It is an engine that is able to make ~/llamay $ llamay run -m qwen2.5:0.5b -seed 7 -temp 0.7 -n 48 -stats=false -p "A sovereign i nference engine is one that" is able to reason about the world in a way that is consistent with the principles of sovere ignty, such as the principle of non-interference in the affairs of others, and the principle of accountability. It is an engine that is able to make ~/llamay $ # same seed, different thread count — still the same tokens ~/llamay $ llamay run -m qwen2.5:0.5b -seed 7 -temp 0.7 -n 48 -t 1 -stats=false -p "A sovere ign inference engine is one that" is able to reason about the world in a way that is consistent with the principles of sovere ignty, such as the principle of non-interference in the affairs of others, and the principle of accountability. It is an engine that is able to make
Where the architectures stand
Twenty-one pinned GGUF files, measured by the project against llama.cpp. Eight clear the gate, four miss it by a fraction, and two are broken — stated in that order because that is the order it matters in.
| Verdict | Architectures | Against llama.cpp's perplexity |
|---|---|---|
| Passing | qwen2 qwen3 gemma2 gemma3 phi3 gpt2 falcon bert | 0.014% – 0.488% |
| Near | llama stablelm and the mixture | 1.2% – 1.4% |
| Explained | gpt-oss | +2.067%, against a reference whose own error bar is ±44.2 |
| Cause fixed, not re-measured | gptneox phi2 | +32.001% and −3.090% |
| Not run | gemma azmx azmx-code azmx-ocr qwen2moe qwen3moe | no pinned file, or a separate tier |
gptneox and phi2 both failed the tokenizer round
trip on a real file, and both now pass it. A byte-level vocabulary is not
uniformly byte-level: Pythia's carries 23 entries of literal spaces marked
USER_DEFINED, and llamay put them through the byte decoder,
which drops a real space. Pythia additionally declares
tokenizer.ggml.pre = "olmo", a name preFor did not
know, so it fell through to the legacy rules and split the text 17.9% finer
than llama.cpp.
Pythia now makes 29,668 tokens of the matrix's 128 KiB of wikitext, which is llama.cpp's count to the token. What has not happened is a re-run of the perplexity rows, which need the weights and a llama.cpp build. The first number should be expected to move; the second should not, because Phi-2's token counts already agreed exactly and only its decoder was broken.
Perplexity over a file you choose, which is how two quantisations of one model get compared:
Transcript — the whole session as text
~/llamay $ llamay ppl -m qwen2.5:0.5b -f README.md -ctx 512 -chunks 6 window 6/6 ppl 37.6841 model ~/.llamay/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc 870eda801d3ab0515 text 16566 tokens, 6 windows of 512, 1530 scored perplexity 37.6841 mean nll 3.6292 elapsed 45.946s
What this does not cover
- Nothing here ran on a GPU. The device graphs decline the parallel residual and the mixture by name, and the i-quants have no device kernel at all.
llamay quantizewill not write an i-quant. Every i-quant file measured was produced by llama.cpp and only read.- Five architectures have no pinned file — the mixture mechanism is covered by the Mixtral row, which is the same router and the same stacked expert tensors, but the Gemma 1 block and the AZMX files are covered by fixtures and by nothing else.
gpt-oss, with its methods
Kept in full because it is the row most likely to be quoted out of context. Token ids: identical to tiktoken's o200k over all 45,235 of a 173 kB corpus, where llama.cpp differs. Greedy, 24 tokens, at two prompts: all 106 compared characters agree on one, and all 111 on the other. Perplexity over four windows of 512: 437.42 against 428.57, which is +2.07% and outside this page's gate — against llama.cpp's own error bar of ±44.2 on the same measurement.
The mechanism behind a mixture's sensitivity here is worth knowing: where the router's distribution is flat, a summation-order difference of 1e-06 flips which expert wins a near-tie, and then the whole continuation diverges.