Build with llamay

Recipes

Things people actually build, each one complete enough to paste into a terminal. Every recipe runs against a local llamay serve on localhost:11435 and sends nothing anywhere.

llamay pull hf:Qwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf
llamay serve

Which shape is your problem?

1. Pull structured data out of unstructured text

The most common thing anyone builds, and where a schema earns its keep: response_format is enforced while sampling, so the reply parses or the request fails. It never returns prose that merely looks like JSON.

curl -s http://localhost:11435/v1/chat/completions \
  -H 'content-type: application/json' -d '{
  "model": "",
  "messages": [{"role": "user", "content":
    "Invoice 4471 from Acme Ltd, dated 3 March 2026, total 1,284.50 EUR, due in 30 days."}],
  "response_format": {
    "type": "json_schema",
    "json_schema": {"name": "invoice", "schema": {
      "type": "object",
      "properties": {
        "number":   {"type": "string"},
        "supplier": {"type": "string"},
        "date":     {"type": "string"},
        "total":    {"type": "number"},
        "currency": {"type": "string", "enum": ["EUR", "USD", "GBP"]},
        "due_days": {"type": "integer"}
      },
      "required": ["number", "supplier", "total", "currency"]
    }}
  }
}' | jq -r '.choices[0].message.content'

{"number":"4471","supplier":"Acme Ltd","date":"2026-03-03",
 "total":1284.5,"currency":"EUR","due_days":30}

enum is worth more than it looks. The sampler cannot emit a currency outside the three, so a downstream ledger never sees one — the constraint shapes the token distribution rather than being checked afterwards.

The whole of this recipe, run:

~/llamay $ llamay serve -m gemma2:2b -addr localhost:11439
time=2026-09-18T08:51:26.911-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:51:26.911-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=gemma2:2b arch=gemma2 max-loaded=2 keepalive=5m0s params="1.59 GB" context=8192 threads=7
kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ # the document, which is the kind that cannot leave the building
~/llamay $ cat build/demo/work/note.txt
Discharge summary. Patient: Anita Sharma, age 62. Admitted 2026-03-04 with chest pain; tropo
nin normal throughout. Discharged 2026-03-06 on aspirin 75mg daily and atorvastatin 20mg nig
htly. Cardiology follow-up in six weeks.
~/llamay $ # the request, with the schema the answer has to fit
~/llamay $ cat scripts/site/demo_extract.json
{
  "model": "gemma2:2b",
  "max_tokens": 200,
  "temperature": 0,
  "messages": [{ "role": "user", "content": "replaced by the note" }],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "discharge",
      "schema": {
        "type": "object",
        "properties": {
          "patient":        { "type": "string" },
          "age":            { "type": "integer" },
          "admitted":       { "type": "string" },
          "discharged":     { "type": "string" },
          "medications":    { "type": "array", "items": { "type": "string" } },
          "followup_weeks": { "type": "integer" }
        },
        "required": ["patient", "age", "admitted", "discharged",
                     "medications", "followup_weeks"],
        "additionalProperties": false
      }
    }
  }
}
~/llamay $ # put the note in the request, send it, and validate what comes back
~/llamay $ jq --rawfile note build/demo/work/note.txt \
>   '.messages[0].content = "Extract the record.\n\n" + $note' \
>   scripts/site/demo_extract.json \
> | curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/json' -
d @- \
> | jq -r '.choices[0].message.content' | jq
{
  "patient": "Anita Sharma",
  "age": 62,
  "admitted": "2026-03-04",
  "discharged": "2026-03-06",
  "medications": [
    "aspirin 75mg daily",
    "atorvastatin 20mg nightly"
  ],
  "followup_weeks": 6
}
~/llamay $ # nothing left the machine, which for this text is the requirement

2. Classify a backlog without paying for the prompt each time

Ten thousand tickets, one instruction. Send the instruction once, fork the context, and each item pays only for its own tokens. The saving is the prefill, which on a long instruction is most of the work.

import json, urllib.request

BASE = "http://localhost:11435"

def post(path, body):
    r = urllib.request.Request(BASE + path, data=json.dumps(body).encode(),
                               headers={"content-type": "application/json"})
    return json.load(urllib.request.urlopen(r))

RULES = (
    "You label support tickets. Answer with exactly one word from: "
    "billing, bug, feature, account, other. No punctuation, no explanation."
)

SYSTEM = {"role": "system", "content": RULES}

def delete(path):
    urllib.request.urlopen(urllib.request.Request(BASE + path, method="DELETE"))

# The instruction, computed once and kept by the server.
ctx = post("/v1/contexts", {"messages": [SYSTEM]})

tickets = ["card declined again", "app crashes on export",
           "can you add dark mode", "cannot reset my password"]

for t in tickets:
    fid = post("/v1/contexts/" + ctx["id"] + "/fork?n=1", {})["data"][0]["id"]
    out = post("/v1/chat/completions", {
        "model": "", "llamay_session": fid, "max_tokens": 4, "temperature": 0,
        # Start with exactly what the context holds, then the new turn.
        "messages": [SYSTEM, {"role": "user", "content": t}],
    })
    reused = out["usage"]["prompt_tokens_details"]["cached_tokens"]
    print(out["choices"][0]["message"]["content"].strip().ljust(10), t.ljust(26),
          "%d of %d prompt tokens reused" % (reused, out["usage"]["prompt_tokens"]))
    delete("/v1/contexts/" + fid)

billing    card declined again        34 of 44 prompt tokens reused
bug        app crashes on export      34 of 45 prompt tokens reused
dark mode  can you add dark mode      34 of 46 prompt tokens reused
account    cannot reset my password   34 of 45 prompt tokens reused

That is qwen2.5:0.5b, and the third answer is not one of the five words. Add "response_format": {"type": "json_schema", "json_schema": {"name": "label", "schema": {"enum": ["billing", "bug", "feature", "account", "other"]}}} and every answer is one of them — the same model then says "feature" for dark mode, and "bug" for the password reset. A constraint fixes the vocabulary, not the judgement.

A fork is a page-table copy, not a memory copy — see the state engine. Forking a 4,000-token instruction a thousand ways costs a thousand small page tables, not four million tokens of KV.

The retrieval half, without leaving the machine:

~/llamay $ # three strings, two of which are about the same thing
~/llamay $ cat build/demo/work/texts.txt
how do I fork a context
context forking and copy-on-write pages
the best route from Pune to Mumbai
~/llamay $ llamay embed -m all-minilm:latest -sim -f build/demo/work/texts.txt
                                        0      1      2
how do I fork a context             1.000  0.630 -0.038
context forking and copy-on-wri…    0.630  1.000 -0.023
the best route from Pune to Mum…   -0.038 -0.023  1.000
~/llamay $ # 0.630 between the first two, below zero for the third. That is the test.

3. Answer from your own documents

Retrieval, reranking and generation in one process. Start the server with an encoder and a cross-encoder alongside the decoder and nothing leaves the box.

llamay serve \
  -m qwen2.5-7b-instruct-q4_k_m.gguf \
  -embed bge-small-en-v1.5-f16.gguf \
  -rerank bge-reranker-base-f16.gguf

docs = [
    "Refunds are issued to the original payment method within 5 business days.",
    "Our warranty covers manufacturing defects for 24 months from purchase.",
    "Shipping is free on orders above 50 EUR within the EU.",
]
question = "how long does a refund take"

ranked = post("/rerank", {"query": question, "documents": docs, "top_n": 2})
top = [docs[r["index"]] for r in ranked["results"]]

answer = post("/v1/chat/completions", {
    "model": "", "temperature": 0, "max_tokens": 120,
    "messages": [
        {"role": "system", "content":
         "Answer only from the passages given. If they do not say, say so."},
        {"role": "user", "content":
         "Passages:\n- " + "\n- ".join(top) + "\n\nQuestion: " + question},
    ],
})
print(answer["choices"][0]["message"]["content"])

For a folder of documents rather than a list in your code, let the server hold the index: llamay index build once, then POST /v1/retrieve returns passages with the file and byte range each came from, filtered by who is asking. This is the whole script, run against gemma2:2b:

import json, os, urllib.request

BASE = "http://localhost:11435"
KEY = os.environ["LLAMAY_API_KEY"]

def post(path, body):
    req = urllib.request.Request(BASE + path, data=json.dumps(body).encode(), headers={
        "content-type": "application/json", "authorization": "Bearer " + KEY})
    return json.load(urllib.request.urlopen(req))

question = "How often is the pump seal inspected?"
hits = post("/v1/retrieve", {"query": question, "top_k": 2})["data"]

sources = "\n\n".join("[%d] %s" % (i + 1, h["text"].strip()) for i, h in enumerate(hits))
answer = post("/v1/chat/completions", {
    "model": "gemma2:2b", "temperature": 0, "max_tokens": 60,
    "messages": [
        {"role": "system", "content":
         "Answer only from the numbered passages. Cite the passage number in brackets."},
        {"role": "user", "content": sources + "\n\nQuestion: " + question},
    ],
})["choices"][0]["message"]["content"]

print(answer)
for i, h in enumerate(hits):
    print("[%d] %s, bytes %d-%d" % (i + 1, h["source"], h["start"], h["end"]))

[1] Every 500 operating hours.

[1] public/pump.md, bytes 0-200
[2] public/valves.txt, bytes 0-115

Building the index and deciding who sees which passage is on documents and citations.

4. Let the model call your functions

Tool arguments are generated under the tool's own JSON Schema, so a call either fits the signature or is not produced. You never parse a hallucinated argument list.

TOOLS = [{
    "type": "function",
    "function": {
        "name": "search_orders",
        "description": "Find orders for a customer.",
        "parameters": {
            "type": "object",
            "properties": {
                "email":  {"type": "string"},
                "status": {"type": "string", "enum": ["open", "shipped", "refunded"]},
            },
            "required": ["email"],
        },
    },
}]

msgs = [{"role": "user", "content": "any refunded orders for [email protected]?"}]
out = post("/v1/chat/completions", {"model": "", "tools": TOOLS, "messages": msgs})

call = out["choices"][0]["message"]["tool_calls"][0]
args = json.loads(call["function"]["arguments"])   # guaranteed to fit the schema
rows = search_orders(**args)                       # your code

msgs += [
    out["choices"][0]["message"],
    {"role": "tool", "tool_call_id": call["id"], "content": json.dumps(rows)},
]
final = post("/v1/chat/completions", {"model": "", "tools": TOOLS, "messages": msgs})
print(final["choices"][0]["message"]["content"])

And the tool-calling half:

~/llamay $ llamay serve -m gemma2:2b -addr localhost:11439
time=2026-09-18T08:49:46.415-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:46.415-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=gemma2:2b arch=gemma2 max-loaded=2 keepalive=5m0s params="1.59 GB" context=8192 threads=7
kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' -d '{
>   "model": "gemma2:2b",
>   "messages": [{"role":"user","content":"What is the weather in Pune?"}],
>   "tools": [{"type":"function","function":{
>     "name": "get_weather",
>     "parameters": {"type":"object",
>       "properties": {"city":{"type":"string"},
>                      "unit":{"enum":["c","f"]}},
>       "required": ["city","unit"]}}}]
> }' | jq '.choices[0].message.tool_calls'
[
  {
    "function": {
      "arguments": "{\"city\":\"Pune\",\"unit\":\"c\"}",
      "name": "get_weather"
    },
    "id": "call_949aba1576ea",
    "index": 0,
    "type": "function"
  }
]
~/llamay $ # city is a string and unit is one of two letters, because it could not be otherw
ise

5. Redact before anything else sees it

A local server is the only kind you can put in front of a hosted one: the redaction step cannot leak what it is redacting, because it never leaves the machine. Start with the rule-based pass, which needs no model and checks each identifier the way it is issued:

curl -s http://localhost:11435/v1/redact -d '{"text":
  "Sam Ortiz ([email protected], +44 7700 900123) paid with 4111 1111 1111 1111 and reported fault 88213."}'

{"findings":[{"type":"email","start":11,"end":26},{"type":"card_number","start":55,"end":74}],
 "text":"Sam Ortiz ([EMAIL], +44 7700 900123) paid with [CARD_NUMBER] and reported fault 88213."}

Two things were left, and both are honest limits: a name is not a pattern, and that phone number is written in a shape the detector does not take. Follow with a model pass for what rules cannot see — names, addresses, anything in prose — and treat its output as a second opinion, not a guarantee. The kinds and their rules are on removing personal data.

6. Stream to a UI

Server-sent events, in the framing every OpenAI client already parses. Each frame carries a piece of text in delta.content; the last one carries finish_reason and usage, and then data: [DONE].

const res = await fetch("http://localhost:11435/v1/chat/completions", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    model: "", stream: true, max_tokens: 400,
    messages: [{ role: "user", content: "explain a bloom filter" }],
  }),
});

const reader = res.body.getReader();
const dec = new TextDecoder();
let buf = "";
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += dec.decode(value, { stream: true });
  const lines = buf.split("\n");
  buf = lines.pop();                      // keep the partial line
  for (const line of lines) {
    if (!line.startsWith("data:")) continue;
    const payload = line.slice(5).trim();
    if (payload === "[DONE]") return;
    const piece = JSON.parse(payload).choices[0]?.delta?.content;
    if (piece) process.stdout.write(piece);
  }
}

Buffer the tail. A chunk boundary can land mid-line, and a client that splits on newline without keeping the remainder drops a token every few hundred — rarely enough that it looks like the model's fault.

7. Stop at a delimiter

Useful when the model fills a template and the rest of the template is yours. The stop string is not included in the answer, and a sequence split across two tokens still stops.

curl -s http://localhost:11435/v1/chat/completions \
  -H 'content-type: application/json' -d '{
  "model": "", "stop": ["\n\n"], "max_tokens": 200,
  "messages": [{"role": "user", "content": "Write one paragraph about tides."}]
}' | jq -r '.choices[0].finish_reason'

stop

8. Move a warm context to another machine

Build the expensive context once — a codebase, a contract, a manual — write it to a file, and restore it where the work happens. The snapshot carries the model identity, and a restore against different weights is refused rather than producing plausible nonsense.

# on the machine with the corpus: the manual as a system message
SYSTEM=$(jq -n --rawfile m manual.txt '{role: "system", content: $m}')
ID=$(curl -s -X POST localhost:11435/v1/contexts \
      -d "{\"messages\": [$SYSTEM]}" | jq -r .id)
curl -s "localhost:11435/v1/contexts/$ID/snapshot" -o manual.llamayctx

# on another machine, with the same model file
NEW=$(curl -s -X POST localhost:11435/v1/contexts/restore \
        --data-binary @manual.llamayctx | jq -r .id)

# the same system message first, then the question
curl -s localhost:11435/v1/chat/completions -d "{\"model\": \"\", \"llamay_session\": \"$NEW\",
  \"messages\": [$SYSTEM, {\"role\": \"user\", \"content\": \"what does it say about warranty?\"}]}"

Between two running servers, llamay ctx move does this in one command and keeps the context's owner and marking; see moving a context.

9. Run a nightly job over a corpus

Concurrency is the server's business, not yours: requests queue and batch against the same weights. Sending them in parallel fills the device; sending far too many at once only grows the queue.

import asyncio, aiohttp

BASE = "http://localhost:11435/v1/chat/completions"
SEM = asyncio.Semaphore(8)          # match -concurrency on the server

async def one(session, text):
    async with SEM:
        async with session.post(BASE, json={
            "model": "", "temperature": 0, "max_tokens": 160,
            "messages": [
                {"role": "system", "content": "Summarise in one sentence."},
                {"role": "user", "content": text},
            ],
        }) as r:
            d = await r.json()
            return d["choices"][0]["message"]["content"]

async def main(docs):
    async with aiohttp.ClientSession() as s:
        return await asyncio.gather(*(one(s, d) for d in docs))

print(asyncio.run(main(open("corpus.txt").read().split("\n\n"))))

JSON that parses on the first attempt, from a model too small to be trusted to remember a brace:

~/llamay $ # a 0.5B model, asked for structure, with nothing enforcing it
~/llamay $ llamay run -m qwen2.5:0.5b -chat -n 90 -p "Give me a JSON object with keys name,
port and tls for a local server."
Certainly! Below is a JSON object containing the specified keys:

```json
{
  "name": "localServer",
  "port": "8080",
  "tls": "true"
}
```

This object defines a local server with the following properties:
- `name`: The name of the server (e.g., "localServer").
- `port`: The port on which the server will listen (e.g., `8

prompt 25 tokens in 175ms (142.5 tok/s) · generated 90 in 7.357s (12.2 tok/s) · stop: length
~/llamay $ # the same request with -json: the sampler cannot emit an invalid token
~/llamay $ llamay run -m qwen2.5:0.5b -chat -json -n 90 -p "Give me a JSON object with keys
name, port and tls for a local server."
{
  "name": "Local server",
  "port": 8080,
  "tls": true
}

prompt 25 tokens in 199ms (125.7 tok/s) · generated 26 in 1.668s (15.6 tok/s) · stop: eog
json mask: 27 steps, 11.97 ms each, 19.4% of decode · no mask cache in this automaton
~/llamay $ llamay run -m qwen2.5:0.5b -chat -json -n 90 -p "Give me a JSON object with keys
name, port and tls for a local server." | jq .

prompt 25 tokens in 172ms (145.3 tok/s) · generated 26 in 1.473s (17.7 tok/s) · stop: eog
json mask: 27 steps, 11.10 ms each, 20.4% of decode · no mask cache in this automaton
{
  "name": "Local server",
  "port": 8080,
  "tls": true
}

10. Constrain output to a grammar

When a schema is the wrong shape — a query language, a DSL, a fixed vocabulary — GBNF constrains the sampler directly.

cat > move.gbnf <<'G'
root   ::= piece square square
piece  ::= "K" | "Q" | "R" | "B" | "N" | ""
square ::= [a-h] [1-8]
G

llamay run -m qwen2.5:0.5b -grammar-file move.gbnf \
  -p "White's best opening move, in algebraic notation:"

11. Grade your own prompts

The same split llamay uses on itself: deterministic assertions that need no model to judge, and a rubric scored by a stronger one. See Correctness for why the two halves are reported separately.

CASES = [
    {"ask": "is this spam? 'WIN A FREE IPHONE NOW'",   "want": "yes"},
    {"ask": "is this spam? 'your package arrives tuesday'", "want": "no"},
]

passed = 0
for c in CASES:
    out = post("/v1/chat/completions", {
        "model": "", "temperature": 0, "max_tokens": 4,
        "messages": [
            {"role": "system", "content": "Answer yes or no. One word."},
            {"role": "user", "content": c["ask"]},
        ],
    })["choices"][0]["message"]["content"]
    ok = out.strip().lower().startswith(c["want"])
    passed += ok
    print(("ok  " if ok else "FAIL"), c["ask"][:40], "->", out.strip())
print(passed, "/", len(CASES))

Twelve more, in brief

Where to go next

Deployment patterns covers the shapes these recipes sit inside — one machine, a queue, a fleet, an air-gapped rack. By sector is the same material organised by the constraint you work under rather than the thing you are building. The HTTP API is the exhaustive reference for every field used above.