Capabilities

Embeddings and reranking

Retrieval needs an encoder as much as generation needs a decoder, and a deployment that runs one engine for the answer and a Python service for the lookup has not actually been made local. Both stages run in the same process, on the same address, from the same binary.

The two stages

The reranker is where the precision is and the embedder is where the scale is; running both means the expensive one only ever sees twenty pairs.

Embeddings

llamay embed -m all-minilm.gguf -p "the capital of France"
llamay embed -m all-minilm.gguf -f sentences.txt -sim

BERT-family encoders run with bidirectional attention, WordPiece and mean pooling — -pool overrides that with mean, cls or last. Vectors come out unit-length, so a dot product is a cosine similarity.

Over HTTP: POST /v1/embeddings in OpenAI's shape, including the base64 encoding_format its official client asks for by default, and Ollama's POST /api/embed and /api/embeddings. An embedding endpoint reachable only by executing a CLI is not an endpoint; it is a subprocess.

Similarity between three strings, computed on the machine that holds them:

Transcript — the whole session as text

~/llamay $ # three strings, two of which are about the same thing
~/llamay $ cat build/demo/work/texts.txt
how do I fork a context
context forking and copy-on-write pages
the best route from Pune to Mumbai
~/llamay $ llamay embed -m all-minilm:latest -sim -f build/demo/work/texts.txt
                                        0      1      2
how do I fork a context             1.000  0.630 -0.038
context forking and copy-on-wri…    0.630  1.000 -0.023
the best route from Pune to Mum…   -0.038 -0.023  1.000
~/llamay $ # 0.630 between the first two, below zero for the third. That is the test.

Similarity, without leaving the machine. llamay embed -sim prints the matrix, which is the only thing most retrieval work needs and the thing an API bill is usually for. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:08

Reranking

llamay rerank -m ms-marco-MiniLM-L6-v2.gguf \
  -q "what is the capital of France?" -f candidates.txt -top 5

Served at POST /rerank, /v1/rerank and /rerank/v1/rerank, in both shapes RAG stacks emit: Jina's and Cohere's documents/relevance_score, and text-embeddings-inference's texts/score. Whichever key the body carries decides the answer's shape.

Cross-encoders declare themselves badly

Of the three real files this was built against — bge-reranker-v2-m3, bge-reranker-base and ms-marco-MiniLM-L-6-v2 — none writes the pooling_type code that means "rank": two omit the key and one writes 1, which means mean. So llamay honours that declaration when it is there and otherwise recognises the classification head, under either of the two tensor namings in use. Scored against llama.cpp on a real file, ms-marco-MiniLM-L6-v2 agrees to 3.5e-4 of a logit.

An encoder served beside the decoder, on one port:

Transcript — the whole session as text

~/llamay $ llamay serve -m qwen2.5:0.5b -embed all-minilm:latest -addr localhost:11439
time=2026-09-18T08:49:56.609-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:56.624-04:00 level=INFO msg="encoder loaded" model=all-MiniLM-L6-v2 pa
th=~/.llamay/models/blobs/sha256-797b70c4edf85907fe0a49eb85811256f65fa0f7bf52166b147fd16be2b
e4662
time=2026-09-18T08:49:56.624-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/embeddings -H 'content-type: application/json'
\
>   -d '{"model":"all-minilm:latest","input":["a local inference engine","a bicycle repair m
anual"]}' \
>   | jq '{model, n: (.data|length), dims: (.data[0].embedding|length)}'
{
  "model": "all-MiniLM-L6-v2",
  "n": 2,
  "dims": 384
}
~/llamay $ curl -s http://localhost:11439/v1/embeddings -H 'content-type: application/json'
\
>   -d '{"model":"all-minilm:latest","input":["hello"]}' | jq '.data[0].embedding[0:6]'
[
  -0.06279965,
  0.054916486,
  0.05200889,
  0.085709885,
  -0.082876645,
  -0.07456734
]

Embeddings beside the decoder. One process, one port: an encoder served at /v1/embeddings next to the chat model, so a retrieval pipeline is local end to end. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:17

Serving them beside a decoder

# generation and both retrieval stages on one address, in one process
llamay serve -m azmx-one-q4.gguf -embed all-minilm.gguf -rerank ms-marco.gguf

# an encoder on its own; the generation routes refuse rather than 404
llamay serve -m all-minilm.gguf

# a cross-encoder on its own; generation and /v1/embeddings both refuse, by name
llamay serve -m ms-marco-MiniLM-L6-v2.gguf

A route that exists but cannot be served refuses by name rather than answering 404. A 404 says "this server does not have that feature", which sends the caller looking for a different server; a named refusal says "this server has it and was not given an encoder", which sends them to the flag.