Build with llamay

Migrating to llamay

From OpenAI, Anthropic, Ollama or llama.cpp. In most cases the change is a base URL, because llamay speaks those APIs rather than translating them — the differences that remain are listed here rather than discovered.

From the OpenAI API

Change two lines. The SDK does not need to know.

Python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11435/v1",
    api_key="not-used",      # required by the SDK, ignored unless -api-key is set
)

client.chat.completions.create(
    model="",                 # "" means the server's loaded model
    messages=[{"role": "user", "content": "hello"}],
)

TypeScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11435/v1",
  apiKey: "not-used",
});

await client.chat.completions.create({
  model: "",
  messages: [{ role: "user", content: "hello" }],
});

Or set OPENAI_BASE_URL and touch no code at all — which is the version that works for a dependency you do not control. 

export OPENAI_BASE_URL=http://localhost:11435/v1
export OPENAI_API_KEY=not-used

What is different

Fieldllamay
modelA file or a store name. "" means whatever the server loaded.
stopSupported, as a string or a list.
response_formatSupported, and enforced while sampling rather than checked after. A schema cannot come back malformed.
tools, tool_choiceSupported. Arguments are generated under the tool's own schema.
max_tokensSupported; the default is 512. max_completion_tokens is read too, and wins when a client sends both.
nAccepted and ignored. Send the request twice, or fork a context.
logprobs, top_logprobsSupported, up to 20, on non-streaming requests.
logit_bias, presence_penalty, frequency_penaltySupported.
userAccepted and ignored — there is no abuse pipeline to report to.
usageReturned, plus prompt_tokens_details.cached_tokens and llamay's own prefill and decode timings.

Unknown fields are ignored rather than refused. A client that sends something llamay does not implement gets an answer, not a 400. That is deliberate — SDKs send fields by default that you never chose — but it means a typo is silent. If a parameter seems to do nothing, check it against the API reference.

All three dialects answering on the same port, against the same weights:

Transcript — the whole session as text 
~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:13.209-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:13.210-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ # OpenAI
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' \
>   -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"Say hi."}]}' | jq -r '
.choices[0].message.content'
Hello! How can I assist you today?
~/llamay $ # Anthropic
~/llamay $ curl -s http://localhost:11439/v1/messages -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","max_tokens":40,"messages":[{"role":"user","content":"Say hi
."}]}' | jq -r '.content[0].text'
Hello! How can I assist you today?
~/llamay $ # Ollama
~/llamay $ curl -s http://localhost:11439/api/chat -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":false,"messages":[{"role":"user","content":"Say hi.
"}]}' | jq -r '.message.content'
Hello! How can I assist you today?
~/llamay $ curl -s http://localhost:11439/api/tags | jq '.models[].name'
"all-minilm:latest"
"gemma2:2b"
"gemma3:270m"
"gpt2:q8_0"
"qwen2.5:0.5b"
"smollm2:135m"
"azmx-code-test:latest"
Three APIs, one server, one model. OpenAI, Anthropic and Ollama all answer on the same port against the same loaded weights. Nothing is translated at a proxy; the routes are native. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:27

From the Anthropic API

from anthropic import Anthropic

client = Anthropic(base_url="http://localhost:11435", api_key="not-used")

client.messages.create(
    model="", max_tokens=512,
    system="Answer in one sentence.",
    messages=[{"role": "user", "content": "hello"}],
)

/v1/messages is served natively: system as a string or a list of blocks, stop_sequences, tools and tool_choice, streaming in Anthropic's event shape, and stop_reason distinguishing end_turn from stop_sequence with stop_sequence naming which one matched. max_tokens is required there, as it is upstream.

From Ollama

Your models come with you. llamay reads Ollama's store directly, so nothing is re-downloaded and nothing is converted. 

llamay run -m llama3.2:3b -p "hello"      # the model Ollama already pulled
llamay serve                               # /api/chat, /api/generate, /api/tags all served
Ollamallamay
ollama run llama3.2llamay run -m llama3.2
ollama servellamay serve
ollama pull …llamay pull …, or POST /api/pull
ollama listllamay models, or GET /api/tags
ollama rm · ollama cpllamay rm · llamay cp (llamay's own store; Ollama's is only read)
Port 11434Port 11435 — both can run at once
options.num_predictSame, and options.stop works
ModelfileNo equivalent. Put the system prompt in the request, or in a context you fork.

The different port is on purpose: you can point one client at each and compare them on the same machine, against the same weights, without stopping either.

From llama.cpp

Same GGUF files, same quantisation names, same tokenizer behaviour — the correctness page covers what is checked against llama.cpp and what that check actually proves.

llama.cppllamay
llama-server -m model.ggufllamay serve -m model.gguf
-c 8192Not needed — the context length is the model's own, read from the file
-ngl 99Not needed — serve uses the GPU when the build has one; -ngl is accepted as a spelling of -gpu
-np 4-concurrency 4, and -batch for continuous batching
--grammar-file x.gbnf-grammar-file x.gbnf
--lora x.gguf-lora name=x.gguf, served as the model <base>+name
--mmproj x.gguf-mmproj x.gguf, with -gpu=false
-ctk q8_0 -ctv q8_0-kv q8, or -kv vq8 for quantised values with exact keys
SlotsContexts — named, forkable, and writable to a file

The thing with no equivalent upstream is the state engine: a context is a first-class object you can fork N ways as a page-table copy, snapshot to disk, and restore on another machine.

LangChain, LlamaIndex and the rest

Anything that takes an OpenAI-compatible base URL works unchanged.

LangChain

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:11435/v1",
    api_key="not-used", model="",
)

LlamaIndex

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    api_base="http://localhost:11435/v1",
    api_key="not-used", model="",
    is_chat_model=True,
)

For embeddings, point the same base URL at /v1/embeddings and start the server with -embed. One process then holds the decoder, the encoder and the cross-encoder, which is what makes a local retrieval stack a single binary rather than three services.

The official OpenAI client, in a clean virtualenv, pointed at llamay by one environment variable:

Transcript — the whole session as text 
~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:51:03.682-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:51:03.682-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ # the official OpenAI client, in a clean virtualenv
~/llamay $ python3.12 -m venv build/demo/work/venv && build/demo/work/venv/bin/pip install -
q openai \
>   && build/demo/work/venv/bin/python -c 'import openai; print(openai.__version__)'

[notice] A new release of pip is available: 25.0.1 -> 26.2.1
[notice] To update, run: python3.12 -m pip install --upgrade pip
3.15.0
~/llamay $ cat scripts/site/demo_client.py
#!/usr/bin/env python3
"""A script written against OpenAI, run against llamay.

Nothing here knows what it is talking to. `OpenAI()` reads OPENAI_BASE_URL and
OPENAI_API_KEY from the environment, which is the documented way to point the
official client at a compatible server, and llamay is one. This file is in the
repository so the demo on /docs/migrate/ can `cat` the thing it then runs.
"""

import os

from openai import OpenAI

client = OpenAI()

print("base_url:", client.base_url, "\n")

chat = client.chat.completions.create(
    model=os.environ.get("MODEL", "qwen2.5:0.5b"),
    messages=[{"role": "user", "content": "In one sentence: what is a KV cache for?"}],
    max_tokens=60,
)
print("chat:", chat.choices[0].message.content.strip())
print("usage:", chat.usage.prompt_tokens, "prompt +",
      chat.usage.completion_tokens, "completion\n")

print("stream: ", end="", flush=True)
for chunk in client.chat.completions.create(
    model=os.environ.get("MODEL", "qwen2.5:0.5b"),
    messages=[{"role": "user", "content": "Count from one to six."}],
    max_tokens=40,
    stream=True,
):
    piece = chunk.choices[0].delta.content
    if piece:
        print(piece, end="", flush=True)
print("\n")

print("models:", ", ".join(m.id for m in client.models.list().data))
~/llamay $ # one environment variable is the entire migration
~/llamay $ OPENAI_BASE_URL=http://localhost:11439/v1 OPENAI_API_KEY=unused \
>   MODEL=qwen2.5:0.5b build/demo/work/venv/bin/python scripts/site/demo_client.py
base_url: http://localhost:11439/v1/

chat: The Cache (KV) Cache is a cache used to store key-value pairs for quick access to data
.
usage: 19 prompt + 21 completion

stream: Sure, here's the sequence from 1 to 6:

1, 2, 3, 4, 5, 6

You can continue counting by following the pattern of increasing

models: all-minilm:latest, gemma2:2b, gemma3:270m, gpt2:q8_0, qwen2.5:0.5b, smollm2:135m, az
mx-code-test:latest
Changing one line in an existing client. A script written against OpenAI, pointed at llamay by its base URL. The request, the response shape and the streaming format are the ones the client already expects. Recorded 2026-09-18 · Apple M4, macOS 26.0.1 · llamay v0.1.58-1-gb14f550 · metal:Apple M4 · 0:26

A migration checklist

  1. Point the base URL at llamay and run your existing tests. Most will pass.
  2. Pick a model that fits. A 7B is not GPT-4. See sizing, and expect to shorten prompts — prompting local models is about exactly this.
  3. Replace format-by-prose with response_format. Then delete the sentences asking for JSON; they are now competing with instructions that still matter.
  4. Move stable material into the system turn so the prefix cache can hold it, and take the timestamp out of it.
  5. Handle 429 rather than 500. A local server has a finite queue and says so with a Retry-After.
  6. Check [DONE] on streams. A truncated stream and a finished one otherwise look the same.
  7. Decide what happens when it cannot answer. Local-first with hosted overflow is a documented pattern, and the redaction step in the middle only works because it is local.

What to expect, honestly

Quality

A 7B will not match a frontier model on open-ended reasoning. On extraction, classification, routing and rewriting — most of what production actually runs — the gap is much smaller than the benchmarks suggest.

Latency

First token is usually faster: no network. Throughput depends on your hardware, and a warm prefix is the difference between 200 ms and 8 s on a long system prompt.

Cost

Fixed rather than per token. That inverts the economics of anything high volume and low value per item, which is where most disappointing API bills come from.

Operations

You now own uptime, memory and model updates. That is a real cost and the honest trade for the three above.

Where to go next

Prompting local models is the adjustment that matters most after the base URL. Errors is what the new failures mean. Deployment patterns is where it runs once it is more than your laptop.