Operating

Deployment patterns

Nine shapes a llamay deployment takes, what each one is good at, and the failure each one has. Pick by the constraint that actually binds you — latency, isolation, cost per token, or a network that does not exist.

Choosing one

1. Sidecar

llamay runs beside your application, on the same host, bound to loopback. The application talks to localhost:11435 and the port is reachable from nowhere else.

# docker-compose.yml
services:
  app:
    build: .
    environment:
      OPENAI_BASE_URL: http://llamay:11435/v1
      OPENAI_API_KEY: not-used-but-sdks-insist
  llamay:
    build: ./llamay        # the Dockerfile on the install page
    command: ["serve", "-addr", ":11435", "-m", "/models/qwen.gguf", "-pull=false"]
    volumes: ["./models:/models:ro"]
    # No ports: published. Only `app` can reach it, over the compose network.

There is no public llamay image; the install page has a Dockerfile that builds one from the release tarball.

Good at: the simplest thing that works, and the easiest to reason about in a review — there is no second machine to secure. Fails when: your application scales horizontally and every replica loads its own copy of the weights. At that point the weights, not the requests, are what you are paying for.

2. A pool behind a queue

One llamay per GPU, a queue in front, workers pulling from it. The queue is what gives you back-pressure; llamay's own scheduler batches whatever arrives.

Good at: throughput, and surviving a burst without dropping it. Fails when: you put a load balancer in front instead of a queue. Round-robin sends a request to a machine whose prefix cache has never seen it, which is the difference between a 200 ms answer and an 8 s one.

The pool pattern, under eight simultaneous callers:

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439 -batch 8 -concurrency 8
time=2026-09-18T08:49:16.060-04:00 level=INFO msg="device graph not built: -batch schedules
several sequences per step and the graph runs one context per pass, so serving continues on
the CPU with GPU-offloaded matmuls. Drop -batch to decode on the graph"
time=2026-09-18T08:49:16.061-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ time (for i in 1 2 3 4 5 6 7 8; do \
>   curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/json' \
>     -d "{\"model\":\"qwen2.5:0.5b\",\"max_tokens\":48,\"messages\":[{\"role\":\"user\",\"c
ontent\":\"Fact number $i about storage:\"}]}" \
>     | jq -r '.choices[0].message.content' & \
> done; wait)
Fact number 5 about storage is: Data stored in a computer system is known as data storage.
Fact number 1 about storage is:

Most of the storage used in today's computers is in the form of RAM (Random Access Memory).
RAM is volatile, which means that when the computer is turned off, the data stored in RAM is
Fact number 3 about storage is:

Storage capacity refers to the total amount of data that can be stored in a given space or o
n a device. It is often measured in terms of storage units, such as gigabytes (GB), ter
Fact number 2 about storage is:

Many types of storage devices, such as hard drives, solid-state drives (SSDs), and solid-sta
te hard drives (SSHDs), are now available in both external and internal forms, allowing for
Fact number 4 about storage is: Storage can be classified into two main types: hard disk dri
ves (HDDs) and solid-state drives (SSDs). HDDs are used in traditional computers and laptops
, while SSDs are used
Fact number 7 about storage is: "Data is stored in various types of physical and digital sto
rage devices, such as hard drives, solid-state drives (SSDs), magnetic tapes, and optical di
scs, as well as in memory and other
Fact number 6 about storage is: Data stored in a database can be accessed and manipulated us
ing various programming languages, such as SQL, Python, and Java. This allows for efficient
data retrieval and modification, and enables users to perform complex operations
Fact number 8 about storage is: Data can be stored in a variety of physical and digital form
s, including but not limited to:

  * Physical storage: Data can be stored in tangible physical locations such as hard drives,
 tape drives,

real    0m22.953s
user    0m0.063s
sys     0m0.077s
~/llamay $ curl -s http://localhost:11439/v1/stats | jq
{
  "arch": "qwen2",
  "backend": "metal:Apple M4",
  "batching": {
    "active": 0,
    "admitted": 8,
    "batch_avg": 7.270833333333333,
    "completed": 8,
    "decode_steps": 48,
    "decode_tokens": 349,
    "device_bypassed": false,
    "device_prefill": 0,
    "device_rows": 0,
    "prefill_tokens": 120,
    "queued": 0,
    "rejected": 0,
    "steps": 49
  },
  "context": 32768,
  "kv_allocated": 8,
  "kv_bytes": 12582912,
  "kv_freed": 0,
  "kv_pages": 8,
  "layers": 24,
  "model": "Qwen2.5 0.5B Instruct",
  "model_id": "590bc10b059c7189",
  "prefix_entries": 8,
  "served": 8,
  "sessions": 0,
  "threads": 7,
  "uptime_s": 23,
  "weight_bytes": 391859712
}

3. Cache affinity

If you must balance rather than queue, balance on a key that keeps a conversation on the machine that already holds it. Hash the session, not the request.

upstream llamay {
    hash $http_x_session consistent;
    server gpu-1.example.internal:11435;
    server gpu-2.example.internal:11435;
    server gpu-3.example.internal:11435;
}

Good at: preserving the prefix cache, which is most of the latency on a long system prompt. Fails when: a machine dies and every conversation it held reshuffles at once. Consistent hashing limits the blast radius; it does not remove it.

4. Air-gapped

No egress at all. The binary is static with no runtime dependency, so the transfer is a file: the executable, the weights, and a checksum.

# on a connected machine
llamay pull hf:Qwen/Qwen2.5-7B-Instruct-GGUF/qwen2.5-7b-instruct-q4_k_m.gguf
sha256sum llamay ~/.llamay/models/blobs/* > MANIFEST

# carry llamay, the blobs and MANIFEST across; on the far side
sha256sum -c MANIFEST
llamay serve -m qwen2.5-7b-instruct-q4_k_m.gguf -pull=false

-pull=false removes the write routes entirely, so a deployment that is not allowed to fetch anything also cannot be asked to. Fails when: nobody planned how a model gets updated. Write the transfer procedure down before the first one, not after the model is a year old.

The air-gapped pattern is not a configuration — it is what happens when there is nothing to configure:

~/llamay $ # prove there is a network first
~/llamay $ curl -s -o /dev/null -w 'github.com: %{http_code}\n' -m 8 https://github.com
github.com: 200
~/llamay $ # now ask llamay for 80 tokens with the resolver pointed at nothing
~/llamay $ env -i PATH="$PATH" HOME="$HOME" RES_OPTIONS=attempts:0 \
>   llamay run -m qwen2.5:0.5b -chat -n 80 -p "Name one thing a local model can do that a ho
sted one cannot."
A local model can do the following:

- Generate and update content faster and more efficiently than a hosted model.

- Improve the performance and efficiency of web applications by caching and optimizing the m
odel.

- Offer a more personalized experience for users by adapting to their specific needs and pre
ferences.

- Offer better accessibility and scalability for local content distribution and delivery.

prompt 22 tokens in 170ms (129.0 tok/s) · generated 68 in 5.168s (13.2 tok/s) · stop: eog
~/llamay $ # no resolver, no token, no telemetry — the weights were already here

5. Multi-tenant, one process

Contexts are the isolation boundary. A context belongs to the model it was created on and is addressed by an opaque, random id. Give each tenant its own key in -api-keys and a context also belongs to the caller that made it: another tenant's key gets a 404 for it, even with the id.

# one context per tenant, created at sign-in and kept
system = {"role": "system", "content": tenant_rules}
ctx = post("/v1/contexts", {"messages": [system]}, key=tenant_key)
store.put(tenant_id, ctx["id"])

# every later turn continues it: the whole conversation so far, then the new
# turn. The part the context already holds is not computed again.
history.append({"role": "user", "content": turn})
post("/v1/chat/completions", {
    "model": "", "llamay_session": store.get(tenant_id),
    "messages": [system] + history,
}, key=tenant_key)

6. Local first, hosted on overflow

The local model answers what it can; anything it declines or takes too long on goes elsewhere. The interesting part is the redaction step in between, which only works because it is local.

7. On the user's own machine

Ship the engine with your application. No account, no key, no egress, and the feature works with the network off. The cost is distribution size and the first-run download of weights.

The macOS app is this pattern: a static binary, a model store under ~/.llamay, and a UI that talks to loopback. Nothing about it requires llamay to have written the UI.

8. In CI

A job that reviews a diff, drafts release notes or checks documentation has no key to configure and no egress rule to get approved, which is usually what kills these before they ship.

- name: Review the diff
  run: |
    curl -sL https://llamay.com/install.sh | sh
    llamay pull hf:Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf
    llamay serve & sleep 5
    git diff origin/main... | python3 tools/review.py

Fails when: the runner is ephemeral and re-downloads the weights every build. Cache the model directory keyed on the model name, not on the commit.

9. Pre-warmed contexts

If every request shares a large prefix — a manual, a schema, a codebase — build it once at startup and fork per request. The first request after a deploy is otherwise the slowest one anybody sees.

HANDBOOK = {"role": "system", "content": open("handbook.txt").read()}

def boot():
    ctx = post("/v1/contexts", {"messages": [HANDBOOK]})
    app.state.base = ctx["id"]          # built once, before traffic

def handle(question):
    fid = post("/v1/contexts/" + app.state.base + "/fork?n=1", {})["data"][0]["id"]
    try:
        return post("/v1/chat/completions", {
            "model": "", "llamay_session": fid,
            # the handbook first, exactly as the context holds it
            "messages": [HANDBOOK, {"role": "user", "content": question}],
        })
    finally:
        delete("/v1/contexts/" + fid)   # forks are cheap, but they are not free

The base context is itself closed after two hours unused; rebuild it when a fork answers 404, or set -session-ttl -1 to keep it.

Sizing, honestly

The weights have to fit in memory the device can reach, and the KV cache grows with context length times concurrency. A 7B at q4_k_m is roughly 4.5 GB of weights; the cache is the number that surprises people, and vq8 roughly halves it at a cost the verifier measures rather than asserts.

Where to go next

Serving is the operational detail behind these shapes — the scheduler, residency, authentication and what is logged. Recipes is what runs inside them. By sector is the same question asked from the compliance side.