Operating

Serving

llamay serve is one process holding one or more models, a scheduler in front of them, and four API families on one address. This page is what to set and why, rather than a list of flags with their help strings repeated.

Starting it

llamay serve                                  # first decoder in the store, localhost:11435
llamay serve -m gemma3:270m -addr :8080       # a specific model, a specific address
llamay serve -batch 8 -max-loaded 3           # continuous batching, three models resident
llamay serve -m azmx-one-q4.gguf \
  -embed all-minilm.gguf -rerank ms-marco-MiniLM-L6-v2.gguf

With no -m, serve takes the first decoder in the store, sorted by name so a restart serves the same one. A decoder rather than simply the first name, because an encoder-only server refuses every generation route — and on a store holding four decoders and one all-minilm, sorting alone served the encoder. Both llamay's own store and Ollama's are searched, which is the same pair -m resolves against. The model named by -m is pinned, which is what makes a request that names nothing always have an answer.

An empty store is not an error

On a machine with no models yet, serve starts anyway and answers the routes that mean something without one: /healthz, empty /v1/models, /api/tags and /api/ps; /api/version; POST /api/pull; and a /studio page that offers the pull. Everything else answers 503 with the reason and the command that fixes it. The moment a model lands — from that route or from llamay pull in a terminal — the same process loads it and starts serving properly, with no restart.

It matters because every packaged install starts the server with no -m, so "empty store" and "fresh install" are the same moment. This used to exit, and a supervisor that restarts on exit turned that into a loop every three seconds — while the macOS app, which pulls its first model through /api/pull, had no server to pull through.

From nothing to an answered request:

Transcript — the whole session as text 
~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/models | jq
{
  "data": [
    {
      "created": 1789735749,
      "id": "all-minilm:latest",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gemma2:2b",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gemma3:270m",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gpt2:q8_0",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "qwen2.5:0.5b",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "smollm2:135m",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "azmx-code-test:latest",
      "object": "model",
      "owned_by": "azmx"
    }
  ],
  "object": "list"
}
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' \
>   -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"One sentence: why run
a model locally?"}]}' | jq
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "running a model locally can be beneficial because it allows you to optim
ize the model locally, which can be faster and more efficient compared to running the model
globally.",
        "role": "assistant"
      }
    }
  ],
  "created": 1789735751,
  "id": "chatcmpl-llamay",
  "model": "qwen2.5:0.5b",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 31,
    "llamay_cached_prompt_tokens": 0,
    "llamay_decode_ms": 1528.48,
    "llamay_prefill_ms": 153.028,
    "prompt_tokens": 17,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "total_tokens": 48
  }
}
~/llamay $ # the same route with stream:true — this is the wire format
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
 five."}]}' | head -c 520
data: {"choices":[{"delta":{"content":"Count"},"finish_reason":null,"index":0}],"created":17
89735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}

data: {"choices":[{"delta":{"content":"ing"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}

data: {"choices":[{"delta":{"content":" to"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"cha~/llamay $ # and the same
stream read the way a client reads it
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
 five, then stop."}]}' \
>   | python3 scripts/site/demo_sse.py
1
2
3
4
5
An OpenAI endpoint on your own machine. llamay serve, then the request every client already sends — non-streaming, then streaming, with the base URL as the only thing that changed. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:29

The scheduler

ARRIVAL request holds no memory yet BOUNDED QUEUE a channel and a few hundred bytes ADMISSION — CONCURRENCY 2 KV acquired here not at submission Full → 429, immediately the ceiling is the batch width, not the queue depth EACH SCHEDULER STEP DOES ONE OF TWO THINGS Prefill — 256 tokens per step The prefix tree is matched first, so only the new suffix costs anything. What is left is cut into chunks, so a 30,000-token prompt cannot hold every other sequence still while it runs. -prefill-chunk 256 Decode — batched across sequences N generations read the weights once rather than N times, which is where the throughput of a busy server comes from. Batch composition never changes the output; verify asserts that. -batch 0 (A GOROUTINE EACH) OR -batch N
Acquiring the KV context at admission rather than at submission is the design decision that makes the queue safe: a burst of a thousand requests costs a thousand small structs, not a thousand caches.

The flags, by what they decide

serve takes sixty-one flags. Most deployments set a handful. They fall into groups, and each group has a page that explains it; the complete table, recorded from the 0.4.3 binary, is below.

DecidesFlagsRead
Where it listens-addr (default localhost:11435), -corsBrowsers
Which models-m, -max-loaded, -keepalive, -pull, -loraResidency · LoRA
What else it serves-embed, -rerank, -pool, -index, -index-access, -mmproj, -whisper, -studio, -studio-execEmbeddings · Documents · Images · Speech
How much at once-concurrency (2), -batch (0), -prefill-chunk (256), -prefix-entries (32), -graph-tokens, -specThe scheduler
Hardware and memory-gpu (on), -ngl, -t, -kv, -kv-page, -no-repack, -ctx-mem-budget, -ctx-spill-dirBackends · Spill
Contexts-session-ttl (2h), -context-key, -allow-legacy-snapshotsContexts
Who may call-api-key, -api-keys, -oidc-*, -tls-client-ca, -tls-client-idNaming callers
The wire-tls-cert, -tls-keyTLS
What is recorded-audit, -audit-strict, -audit-sync, -audit-syslog, -audit-continue, -capture, -capture-key, -capture-retainAudit · Capture
Need to know-classificationMarkings
All of the above, checked-profile enterpriseThe profile
Watching it-v, -otlp, -tracesObservability
Constraining every request-grammar, -grammar-file, -lexiconGrammars
The GPU is on by default for serve

serve runs the model on a GPU when the build has one and the model's shapes are supported, and falls back to the CPU otherwise. -gpu=false forces the CPU, which image requests need. run and the other one-shot commands default the other way.

Eight requests arriving together, decoded together:

Transcript — the whole session as text 
~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439 -batch 8 -concurrency 8
time=2026-09-18T08:49:16.060-04:00 level=INFO msg="device graph not built: -batch schedules
several sequences per step and the graph runs one context per pass, so serving continues on
the CPU with GPU-offloaded matmuls. Drop -batch to decode on the graph"
time=2026-09-18T08:49:16.061-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ time (for i in 1 2 3 4 5 6 7 8; do \
>   curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/json' \
>     -d "{\"model\":\"qwen2.5:0.5b\",\"max_tokens\":48,\"messages\":[{\"role\":\"user\",\"c
ontent\":\"Fact number $i about storage:\"}]}" \
>     | jq -r '.choices[0].message.content' & \
> done; wait)
Fact number 5 about storage is: Data stored in a computer system is known as data storage.
Fact number 1 about storage is:

Most of the storage used in today's computers is in the form of RAM (Random Access Memory).
RAM is volatile, which means that when the computer is turned off, the data stored in RAM is
Fact number 3 about storage is:

Storage capacity refers to the total amount of data that can be stored in a given space or o
n a device. It is often measured in terms of storage units, such as gigabytes (GB), ter
Fact number 2 about storage is:

Many types of storage devices, such as hard drives, solid-state drives (SSDs), and solid-sta
te hard drives (SSHDs), are now available in both external and internal forms, allowing for
Fact number 4 about storage is: Storage can be classified into two main types: hard disk dri
ves (HDDs) and solid-state drives (SSDs). HDDs are used in traditional computers and laptops
, while SSDs are used
Fact number 7 about storage is: "Data is stored in various types of physical and digital sto
rage devices, such as hard drives, solid-state drives (SSDs), magnetic tapes, and optical di
scs, as well as in memory and other
Fact number 6 about storage is: Data stored in a database can be accessed and manipulated us
ing various programming languages, such as SQL, Python, and Java. This allows for efficient
data retrieval and modification, and enables users to perform complex operations
Fact number 8 about storage is: Data can be stored in a variety of physical and digital form
s, including but not limited to:

  * Physical storage: Data can be stored in tangible physical locations such as hard drives,
 tape drives,

real    0m22.953s
user    0m0.063s
sys     0m0.077s
~/llamay $ curl -s http://localhost:11439/v1/stats | jq
{
  "arch": "qwen2",
  "backend": "metal:Apple M4",
  "batching": {
    "active": 0,
    "admitted": 8,
    "batch_avg": 7.270833333333333,
    "completed": 8,
    "decode_steps": 48,
    "decode_tokens": 349,
    "device_bypassed": false,
    "device_prefill": 0,
    "device_rows": 0,
    "prefill_tokens": 120,
    "queued": 0,
    "rejected": 0,
    "steps": 49
  },
  "context": 32768,
  "kv_allocated": 8,
  "kv_bytes": 12582912,
  "kv_freed": 0,
  "kv_pages": 8,
  "layers": 24,
  "model": "Qwen2.5 0.5B Instruct",
  "model_id": "590bc10b059c7189",
  "prefix_entries": 8,
  "served": 8,
  "sessions": 0,
  "threads": 7,
  "uptime_s": 23,
  "weight_bytes": 391859712
}
Eight requests at once. Continuous batching: eight generations enter the scheduler, decode together, and leave when each one is done rather than when the slowest is. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:20

Model residency

The server loads models on demand. A request naming any model this machine has loads it, evicting one that has gone idle. 

llamay serve -m qwen2.5:0.5b -max-loaded 3 -keepalive 5m
curl localhost:11435/api/chat -d '{"model":"gemma3:270m","messages":[…]}'   # loads it
curl localhost:11435/api/ps                                                 # what is resident

Two properties are worth stating because they are the ones a naive implementation gets wrong. A model is never unloaded while a request is using it — eviction counts references rather than trusting the idle timer to be slower than a generation. And concurrent requests for the same unloaded model produce one load, not one each.

Authentication

-api-key, or LLAMAY_API_KEY, requires a key on every route but /healthz — which stays open because the thing reading it is usually a container probe with no way to carry a secret.

The environment is the better of the two

A key on the command line is visible in the process list to everyone on the machine. And without a key the server is open, which is the right default for something bound to localhost and the wrong one the moment -addr :11435 is typed — along with -pull=false, since anyone who can reach /api/pull can make the process fetch a URL and spend the machine's disk on it.

A key, and the 401 that is the point of having one:

Transcript — the whole session as text 
~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439 -api-key demo-key-not-a-secret
time=2026-09-18T08:49:40.239-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:40.239-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ # no key: refused
~/llamay $ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:11439/v1/models
401
~/llamay $ curl -s http://localhost:11439/v1/models | jq
{
  "error": {
    "message": "this server requires an API key: send Authorization: Bearer <key>",
    "type": "Unauthorized"
  }
}
~/llamay $ # the health check stays open, because a load balancer has no key
~/llamay $ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:11439/healthz
200
~/llamay $ # with the key
~/llamay $ curl -s http://localhost:11439/v1/models -H 'authorization: Bearer demo-key-not-a
-secret' | jq '.data[].id'
"all-minilm:latest"
"gemma2:2b"
"gemma3:270m"
"gpt2:q8_0"
"qwen2.5:0.5b"
"smollm2:135m"
"azmx-code-test:latest"
A key, and what happens without one. -api-key closes every route but /healthz. The demo is the 401, because that is the one people deploy without checking. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:19

Health, metrics and traces

RouteWhat it answers
GET /healthz{"status":"ok","model":…,"id":…}. This is identity, not liveness — a 200 from a port tells you something is listening, not what.
GET /metricsPrometheus, including the scheduler's own counters: queue depth, admissions, prefill and decode tokens, evictions.
GET /v1/statsThe same numbers as JSON.
OTLPSpans for the request, the queue wait, prefill and decode, joining the caller's traceparent, on both the sequential and the -batch path.

Every flag

Recorded from llamay serve -h in the 0.4.3 binary.

FlagDefaultMeaning
-addr string"localhost:11435"listen address
-allow-legacy-snapshotsrestore and import contexts made before they recorded the sha256 of the weights, even under -profile enterprise or classification markings, where they are otherwise refused (they are checked by layout only, which a fine-tune shares with its base)
-api-key stringrequire this key on every route but /healthz; also read from LLAMAY_API_KEY
-api-keys id:keyfile of id:key lines, one caller each; supersedes -api-key and names the caller in every audit record
-audit fileappend a hash-chained record of every request to this file; check it with llamay audit verify
-audit-continue stringwhen -audit names a new file, continue the chain from this previous one, so a rotation keeps the join; llamay audit verify old new checks both
-audit-strictstop serving if an audit record cannot be written, rather than logging and continuing
-audit-syncsync each audit record to disk as it is written, when its request ends; a streamed response has been delivered by then
-audit-syslogalso send every -audit record to this machine's syslog daemon (authpriv, tag llamay-audit), for a SIEM to collect
-batch intsequences to decode together; 0 runs each generation on its own
-capture stringkeep an encrypted copy of every request and response in this directory, linked to its -audit record; needs -capture-key
-capture-key stringthe records office's public key (llamay capture keygen); the server can write captures and cannot read them
-capture-retain stringdelete capture files older than this, e.g. 7y or 90d; a LEGAL-HOLD file in the directory suspends deletion (default: keep everything)
-classification stringenforce classification markings from this file: levels, a default, and each caller's clearance and compartments (see the security docs)
-concurrency int2generations allowed to run at once
-context-key stringencrypt every snapshot this server hands out with this key file (64 hex characters; openssl rand -hex 32), and restore snapshots sealed under it
-cors stringcomma-separated origins allowed to call this server from a browser, e.g. https://llamay.com
-ctx-mem-budget stringKV memory to keep resident per model, e.g. 8G; past it idle contexts are written to disk and read back on their next use. Empty keeps every context in memory
-ctx-spill-dir stringwhere -ctx-mem-budget writes idle contexts (default $TMPDIR/llamay-spill); the files are encrypted, under -context-key or a key that lives only as long as the process
-embed stringencoder model to serve at /v1/embeddings beside the decoder
-gputruerun the layer stack on the GPU when one is available; -gpu=false forces the CPU
-grammar stringconstrain output to this GBNF grammar, given inline
-grammar-file stringconstrain output to the GBNF grammar in this file
-graph-tokens int256prompt tokens per device prefill pass; wider tiles amortise the encode, and the buffers scale with it
-index stringa document index (llamay index build) to answer POST /v1/retrieve from; needs -embed with the encoder it was built with, and -context-key if it is sealed
-index-access caller: label, ...file of caller: label, ... lines granting callers the index labels they may see; unlabelled passages are visible to every caller
-keepalive duration5m0show long an idle model stays loaded; 0 keeps every loaded model forever
-kv string"f32"KV cache precision: f32, vq8 (values quantised, keys exact), or q8 (both)
-kv-page int64positions per KV page
-lexicon stringnewline-separated word list for lexicon-constrained decoding
-lora <base>+<name>serve a LoRA adapter as the model <base>+<name>: name=adapter.gguf (repeatable)
-m stringpath to a GGUF model (required)
-max-loaded int2models kept in memory at once beside the default; a request for another evicts an idle one, if it must, and then loads it
-max-memory stringmodel weights kept in memory at once, the default's included, e.g. 10G; 0 is no limit. Unset, it is 70% of RAM on a Mac and the memory available at startup elsewhere
-mmproj stringthe model's vision projector (an mmproj GGUF), so chat requests may carry images; images are read on the CPU path
-ngl int-1llama.cpp spelling of -gpu: 0 is CPU only, anything at or above the layer count is the whole model on the GPU
-no-repackkeep weights in their file format instead of rewriting them for a faster kernel (on by default for the CPU and unified memory, off for a discrete GPU)
-oidc-audience stringthe audience tokens must be issued for (your app registration's id or api:// URI)
-oidc-azp stringcomma-separated authorized parties a token's azp must be one of -- what Clerk session tokens carry instead of an audience
-oidc-claim stringpreferred_username; sub for Clerk, email or upn elsewherethe claim that names the caller
-oidc-issuer stringaccept sign-in tokens from this directory (Entra ID, Okta, Keycloak, Clerk...): its issuer URL, exactly as tokens carry it
-oidc-jwks stringthe issuer's signing keys: a JWKS file (works offline), or a URL; default is discovery from -oidc-issuer
-otlp stringOTLP/HTTP collector base URL for traces, e.g. http://localhost:4318; also read from OTEL_EXPORTER_OTLP_ENDPOINT
-pool stringoverride the encoder's pooling: mean, cls or last
-prefill-chunk int256prompt tokens prefilled per scheduler step, so a long prompt does not stall decodes
-prefix-entries int32how many shared prefixes to keep cached
-profile string"personal"startup rules: personal (none), or enterprise (keys and TLS on a reachable address, and a strict, synced -audit record)
-pulltruelet POST /api/pull, /api/copy and DELETE /api/delete write to the model store
-rerank stringcross-encoder model to serve at /rerank beside the decoder
-session-ttl duration2h0m0show long a context nobody is using is kept; negative keeps every context until it is deleted
-spec string"off"speculative decoding: off, or ngram to draft from the prompt and what has been generated
-studio stringserve the Studio coding surface at /studio over this directory
-studio-execlet Studio run commands in the -studio directory; needs -studio
-t intworker threads (0 picks a default that avoids efficiency cores)
-tls-cert stringserve HTTPS with this PEM certificate (chain first, leaf at the top); needs -tls-key
-tls-client-ca stringrequire every client to present a certificate issued by an authority in this PEM file (mutual TLS)
-tls-client-id stringname each caller by its verified client certificate: cn (e.g. a CAC or PIV card) or email; needs -tls-client-ca
-tls-key stringthe PEM private key for -tls-cert
-tracesrecord spans for queue wait, prefill and decode; implied by -otlp, and logged at debug level without one
-vlog every request
-whisper stringa Whisper speech-to-text model (whisper.cpp's ggml-*.bin) to serve at /v1/audio/transcriptions, and for Studio's microphone

Running it as llama-server

The binary reads the name it was started as: called llamay-server, it runs serve with no subcommand, so deployment tooling written for llama-server needs no change. The Mac app's Install Command Line… links both names; elsewhere, make the link yourself: 

ln -s "$(command -v llamay)" /usr/local/bin/llamay-server
llamay-server -m qwen2.5:0.5b -addr localhost:8080