Reference

HTTP API reference

Every route llamay serve answers, on one address: OpenAI's, Anthropic's and Ollama's, MCP, and llamay's own. For each: what it takes, what it returns, and the flag it needs. Responses shown were returned by llamay 0.3.0.

Basics

Every route

POST /v1/chat/completions

curl -s localhost:11435/v1/chat/completions -d '{"model": "qwen2.5:0.5b", "stream": true, "max_tokens": 3,
  "messages": [{"role": "user", "content": "hi"}]}'

data: {"choices":[{"delta":{"content":"你好"},"finish_reason":null,"index":0}],"created":1790483659,"id":"chatcmpl-6b78080731dd6c957db68a77","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}

data: {"choices":[{"delta":{"content":"！"},"finish_reason":null,"index":0}], ...}

data: {"choices":[{"delta":{"content":"有什么"},"finish_reason":null,"index":0}], ...}

data: {"choices":[{"delta":{},"finish_reason":"length","index":0}], ..., "usage":{"completion_tokens":3,"prompt_tokens":9,"prompt_tokens_details":{"cached_tokens":3},"total_tokens":12, ...}}

data: [DONE]

usage adds prompt_tokens_details.cached_tokens (how much of the prompt was reused) and llamay's llamay_prefill_ms, llamay_decode_ms and llamay_cached_prompt_tokens. finish_reason is stop, length or tool_calls.

A logprobs answer, for the prompt "The capital of France is":

"logprobs": {"content": [{"token": "Paris", "logprob": -0.37225074, "bytes": [80, 97, 114, 105, 115],
  "top_logprobs": [{"token": "Paris", "logprob": -0.37225074, ...},
                   {"token": "The",   "logprob": -1.4470932, ...},
                   {"token": " Paris", "logprob": -4.4920826, ...}]}]}

The OpenAI routes, against a server started thirty seconds earlier:

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:09.326-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/models | jq
{
  "data": [
    {
      "created": 1789735749,
      "id": "all-minilm:latest",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gemma2:2b",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gemma3:270m",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "gpt2:q8_0",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "qwen2.5:0.5b",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "smollm2:135m",
      "object": "model",
      "owned_by": "azmx"
    },
    {
      "created": 1789735749,
      "id": "azmx-code-test:latest",
      "object": "model",
      "owned_by": "azmx"
    }
  ],
  "object": "list"
}
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' \
>   -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"One sentence: why run
a model locally?"}]}' | jq
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "running a model locally can be beneficial because it allows you to optim
ize the model locally, which can be faster and more efficient compared to running the model
globally.",
        "role": "assistant"
      }
    }
  ],
  "created": 1789735751,
  "id": "chatcmpl-llamay",
  "model": "qwen2.5:0.5b",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 31,
    "llamay_cached_prompt_tokens": 0,
    "llamay_decode_ms": 1528.48,
    "llamay_prefill_ms": 153.028,
    "prompt_tokens": 17,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "total_tokens": 48
  }
}
~/llamay $ # the same route with stream:true — this is the wire format
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
 five."}]}' | head -c 520
data: {"choices":[{"delta":{"content":"Count"},"finish_reason":null,"index":0}],"created":17
89735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}

data: {"choices":[{"delta":{"content":"ing"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"chat.completion.chunk"}

data: {"choices":[{"delta":{"content":" to"},"finish_reason":null,"index":0}],"created":1789
735751,"id":"chatcmpl-llamay","model":"qwen2.5:0.5b","object":"cha~/llamay $ # and the same
stream read the way a client reads it
~/llamay $ curl -sN http://localhost:11439/v1/chat/completions -H 'content-type: application
/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":true,"messages":[{"role":"user","content":"Count to
 five, then stop."}]}' \
>   | python3 scripts/site/demo_sse.py
1
2
3
4
5

POST /v1/completions

prompt continued as raw text, with the same sampling fields as chat. logprobs: 5 (a number) asks for that many alternatives, answered in the completions API's own shape: tokens, token_logprobs, top_logprobs (one object of token to log-probability per position) and text_offset (characters into the returned text). A token that is only part of a character is written bytes:\xNN, as OpenAI writes it.

POST /v1/responses

curl -s localhost:11435/v1/responses -d '{"model": "qwen2.5:0.5b", "input": "Say hi.", "max_output_tokens": 8}'

{"created_at":1790483666,"id":"resp_b6ab96ea808f4bbf00196d3d498cc039","model":"qwen2.5:0.5b","object":"response",
 "output":[{"content":[{"annotations":[],"text":"Hello! How can I assist you today","type":"output_text"}],
   "id":"msg_b6ab96ea808f4bbf00196d3d498cc039","role":"assistant","status":"completed","type":"message"}],
 "status":"completed","usage":{"input_tokens":11,"input_tokens_details":{"cached_tokens":0},"output_tokens":8,"total_tokens":19}}

• Built over contexts: previous_response_id continues the previous response's context, so its prompt is not read again.
• input is a string or a list of messages; instructions, max_output_tokens (default 1024), stop, stream and the sampling fields work. store: false drops the context with the record.
• Every turn is a response of its own, with its own id. A turn made with previous_response_id keeps the whole conversation before it, so chaining through any turn continues from that turn.
• The last 256 responses are kept. GET /v1/responses/{id} returns one and DELETE releases it, on whichever model made it. On a server with named callers only the caller who made a response can read or delete it; to anybody else it is a 404.
• Function tools, tool_choice and parallel_tool_calls work as they do on chat completions, with function_call items in output; text.format takes a JSON schema the way response_format does.

POST /v1/embeddings

curl -s localhost:11435/v1/embeddings -d '{"model": "all-minilm", "input": ["a cat sat on the mat"]}'

input is a string or a list of up to 2048 strings (token arrays are refused, and so is an empty string: there is nothing to embed). encoding_format is float or base64, which the OpenAI SDK asks for by default. Vectors are unit length. Needs -embed, or an encoder as -m. Embeddings

POST /v1/audio/transcriptions

curl -s localhost:11435/v1/audio/transcriptions -F [email protected] -F model=whisper-1 -F response_format=json

{"text":"And so my fellow Americans ask not what your country can do for you, ask what you can do for your country."}

Multipart form: file (WAV), model, language, response_format (json, text, verbose_json, srt, vtt), temperature (only 0), and timestamp_granularities[]=segment with verbose_json, whose segments carry Whisper's own start and end times (word is refused). /v1/audio/translations takes the same and writes English. Needs -whisper. Speech to text

GET /v1/models

Every model the server could load — llamay's store and Ollama's, LoRA adapters as <base>+<name>, and the -embed encoder, listed once — as {"object": "list", "data": [{"id", "object", "created", "owned_by", "capabilities"}]}. The same set as /api/tags.

capabilities says what each entry can be asked for: ["chat"] for a model that generates and has a chat template, ["completion"] for a base model with none (it continues text, through /v1/completions), ["embeddings"] for an encoder, and [] for a file in the store llamay cannot run (a vision projector, an architecture it declines). A client building a model picker should offer the entries that include chat.

GET /v1/models/{id} returns one entry of that list — what the OpenAI and Anthropic SDKs call for models.retrieve — matched the way a request's model is, so all-minilm finds all-minilm:latest. A name the list does not hold is a 404.

POST /v1/messages

Anthropic's Messages API: system as a string or blocks, messages with text, image and document blocks, max_tokens (default 1024), stop_sequences, tools and tool_choice (including disable_parallel_tool_use), and streaming in Anthropic's event shape. stop_reason is end_turn, stop_sequence (with stop_sequence naming which), max_tokens or tool_use. Reuse is reported as usage.cache_read_input_tokens. Server-side tool types are refused; logprobs is not supported here. A closing assistant message is a prefill, as in Anthropic's API: the reply continues from its last character, and content holds only what follows.

POST /v1/messages/count_tokens

curl -s localhost:11435/v1/messages/count_tokens -d '{"model": "qwen2.5:0.5b", "messages": [{"role": "user", "content": "Say hi."}]}'

{"input_tokens":11,"llamay_model":"qwen2.5:0.5b"}

Counted by the model's own tokenizer over the rendered prompt — chat template, system and tool definitions included — which is what is actually sent.

Ollama routes

curl -s localhost:11435/api/chat -d '{"model": "qwen2.5:0.5b", "stream": false,
  "options": {"temperature": 0, "num_predict": 5}, "messages": [{"role": "user", "content": "Say hi."}]}'

{"created_at":"2026-09-27T04:34:26.130128Z","done":true,"done_reason":"length","eval_count":5,"eval_duration":222750000,
 "load_duration":0,"message":{"content":"Hello! How can I","role":"assistant"},"model":"qwen2.5:0.5b",
 "prompt_eval_count":11,"prompt_eval_duration":91082000,"total_duration":314062916}

Only llamay_session of llamay's extensions is read on Ollama routes; constraints come from format.

Pulling, copying and deleting

POST /api/pull takes {"model": "<source>"} in any form llamay pull accepts and streams progress as Ollama's clients expect ("stream": false for one answer). POST /api/copy (source, destination) and DELETE /api/delete (model) change names in the store. All three are off under -pull=false (501), and refuse a web page whose origin is not in -cors.

Refused with 501, by name: /api/push (llamay publishes nothing), and /api/create and /api/blobs (no Modelfile parser).

Three dialects, one server, one loaded copy of the weights:

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:13.209-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:13.210-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ # OpenAI
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' \
>   -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"Say hi."}]}' | jq -r '
.choices[0].message.content'
Hello! How can I assist you today?
~/llamay $ # Anthropic
~/llamay $ curl -s http://localhost:11439/v1/messages -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","max_tokens":40,"messages":[{"role":"user","content":"Say hi
."}]}' | jq -r '.content[0].text'
Hello! How can I assist you today?
~/llamay $ # Ollama
~/llamay $ curl -s http://localhost:11439/api/chat -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":false,"messages":[{"role":"user","content":"Say hi.
"}]}' | jq -r '.message.content'
Hello! How can I assist you today?
~/llamay $ curl -s http://localhost:11439/api/tags | jq '.models[].name'
"all-minilm:latest"
"gemma2:2b"
"gemma3:270m"
"gpt2:q8_0"
"qwen2.5:0.5b"
"smollm2:135m"
"azmx-code-test:latest"

POST /rerank

Also /v1/rerank and /rerank/v1/rerank. Needs -rerank. Two request shapes, and the answer follows the one you sent:

{"query": "how long does a refund take", "documents": ["...", "..."], "top_n": 2}
→ {"model": "...", "results": [{"index": 0, "relevance_score": 0.97, "score": 3.4, "document": {"text": "..."}}], "usage": {...}}

{"query": "how long does a refund take", "texts": ["...", "..."], "raw_scores": false}
→ [{"index": 0, "score": 0.97}, ...]

The first is Jina's and Cohere's shape; the second is text-embeddings-inference's. top_n keeps the best that many, after the sort; top_k, an older spelling some clients send, is read only when top_n is not. Reranking

POST /v1/retrieve

{"query": "…", "top_k": 5, "rerank": true} → {"object": "list", "model", "index_model", "data": [{"source", "start", "end", "text", "labels", "score", "rerank_score"}]}. Needs -index and -embed; filtered by the caller's labels. top_k is at most 50. Documents

POST /v1/redact

{"text": "…", "types": ["email", …]} → {"text": "… [EMAIL] …", "findings": [{"type", "start", "end"}]}. types is optional. Redaction

POST /v1/route

What kind of message this is and which chat model on this server should answer it: the Auto choice in every llamay app. It classifies and ranks; it loads no chat model and generates nothing. {"messages": […]} or {"text": "…"}, with "images": n or "has_image": true for attached pictures and "prefer": "local" | "best" | "cheap" → {"object": "route", "category", "confidence", "method", "reason", "model", "model_reason", "vision", "fits", "fits_memory", "memory_budget_bytes", "context", "estimated_tokens", "encoder", "candidates": [{"model", "score", "reason", "params", "context", "vision"}]}.

category is one of image, code, multilingual, reasoning, long-context, quick-chat, general. method says what decided it: attachment (a picture), length (over 6,000 characters), embedding (the trained classifier, 87% on held-out prompts) or rules (keywords, when there is no encoder). The classifier needs all-minilm: the server's -embed encoder when it is that file, otherwise the store's copy, loaded on the first route (llamay pull all-minilm). fits is false when the conversation and room for an answer exceed the chosen model's context, or when the model does not fit in memory; fits_memory says which, and memory_budget_bytes, when the budget is known, is the room the chosen model has beside the others already loaded. "cheap" takes the smallest model that scores within a point of the best; every model here is local, so "local" and "best" are the same ranking. With no chat model installed, model is "".

curl -s localhost:11435/v1/route -d '{"text": "why does my go test hang when I close the channel twice"}'
# {"category": "code", "confidence": 0.97, "method": "embedding",
#  "model": "qwen2.5-coder:7b", "model_reason": "code-tuned; 7B parameters", ...}

POST /v1/systemone

Ollama's decision API (Ollama 0.35), on any chat model this server runs. Send text as state and a set of named questions, each with a closed set of answers; every answer comes back as probabilities, and nothing is generated. The request, the answers and the errors are Ollama's, so its Python client's systemone() and TypeSafe's system_one() work against this server unchanged.

curl -s localhost:11435/v1/systemone -d '{
  "model": "qwen2.5:0.5b",
  "state": {"ticket": "I was charged twice. Please refund the extra payment."},
  "questions": {
    "team": {"type": "choice", "instructions": "Which team should handle this ticket?",
             "criteria": {"billing": "Payments and refunds", "technical": "Bugs and integrations", "other": "None of the above"}},
    "refund": {"type": "noul", "instructions": "Does the customer explicitly ask for a refund?"},
    "urgency": {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["Routine", "Soon", "Urgent"]}
  }}'

{"model": "qwen2.5:0.5b",
 "answers": {
   "team": {"type": "choice", "choice": "billing",
            "probabilities": {"billing": 0.661, "technical": 0.291, "other": 0.048}, "confidence": 0.292},
   "refund": {"type": "noul", "noul": 0.561},
   "urgency": {"type": "score", "score": 1.373, "legend": {"0": "Routine", "1": "Soon", "2": "Urgent"},
               "probabilities": {"0": 0.257, "1": 0.112, "2": 0.630}, "confidence": 0.194}},
 "usage": {"input_tokens": 351, "output_tokens": 0}}

Each question is asked with its options lettered A, B, C…, and its answer is the model's next-token probability for each letter, renormalised over the options: 2 to 26 options per question. confidence is one minus the normalised entropy of those probabilities: 1 when all of it is on one option, 0 when it is spread evenly. input_tokens counts every question's whole prompt, as Ollama does; output_tokens is 0, because nothing is generated.

The questions share the state, so the server reads it once and forks that context for each question, which then costs only its own tokens: with a 300-token ticket, three questions took a third of the time of reading each one whole and eight took a fifth. The state is kept in the prefix cache, so asking more about it later starts from there. X-Llamay-Evaluated-Tokens says how many positions the model actually read. A model with no chat template answers 400, as does a request whose question does not fit the model's context.

Ollama's own decision models, nimble and tev1, are Qwen 3.5 hybrids, which llamay does not run; the answers here are a general chat model's, and a small one is a rough judge. On qwen2.5:0.5b the team for a ticket and the direction of a review's sentiment were usually right, and whether a comment was abusive was not.

Contexts

A context object is {"id", "parent", "created", "tokens", "pages", "shared_pages"}, plus "spilled": true when it is on disk. Every route on an existing context takes ?model=, and POST /v1/contexts takes model in its body; a context belongs to its model and, on a server with named callers, to its caller.

A context in use by another request answers 409; one that does not exist (or is not yours) answers 404. Contexts

A context created, listed and forked over HTTP:

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:07.893-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:07.894-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ curl -s http://localhost:11439/v1/contexts -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","prompt":"You are a careful code reviewer."}' | jq
{
  "created": 1789735748,
  "id": "ctx_1",
  "pages": 1,
  "parent": "",
  "shared_pages": 0,
  "tokens": 7
}
~/llamay $ curl -s http://localhost:11439/v1/contexts | jq
{
  "data": [
    {
      "created": 1789735748,
      "id": "ctx_1",
      "pages": 1,
      "parent": "",
      "shared_pages": 0,
      "tokens": 7
    }
  ]
}
~/llamay $ # fork the one that exists, four ways
~/llamay $ ID=$(curl -s http://localhost:11439/v1/contexts | jq -r '.data[0].id') && \
>   curl -s http://localhost:11439/v1/contexts/$ID/fork -H 'content-type: application/json'
\
>   -d '{"n":4}' | jq
{
  "data": [
    {
      "created": 1789735748,
      "id": "ctx_2",
      "pages": 1,
      "parent": "ctx_1",
      "shared_pages": 1,
      "tokens": 7
    }
  ],
  "fork_ms": 0.009
}
~/llamay $ curl -s http://localhost:11439/v1/contexts | jq '.data | length'
2

POST /mcp

The Model Context Protocol, so an MCP host can use this machine's models without learning the routes above. Protocol version 2026-07-28; each request carries the MCP-Protocol-Version and Mcp-Method headers, and Mcp-Name for tools/call and prompts/get (the name) and resources/read (the URI).

curl -s localhost:11435/mcp -H 'MCP-Protocol-Version: 2026-07-28' -H 'Mcp-Method: tools/list' \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

Four tools: chat (messages, model, max_tokens, temperature, json_schema), complete, embed and list_models. A call answers with the corresponding OpenAI route's JSON as text. Only a browser origin on this machine may call it. resources/list lists the caller's own contexts as llamay://contexts/<id>, and resources/read returns one as the object GET /v1/contexts describes it with; a context that is not the caller's does not exist.

Health and metrics

Observability has the fields and metric names.

llamay's extensions

Accepted on the OpenAI and Anthropic generation routes (Ollama routes read only llamay_session):

On the way back: usage.llamay_cached_prompt_tokens, llamay_prefill_ms and llamay_decode_ms; the X-Llamay-Media header (and llamay_media in a non-streaming OpenAI chat body) when images or video frames were thinned; X-Llamay-Transcribe-Ms on transcriptions.