Build with llamay

Integrating llamay

What it takes to put llamay behind something you have already built. In most cases the whole migration is a base URL, because llamay answers the request shapes your client already sends.

The shape of the work

1 · Decide where it runs

There are three answers and they are not exclusive — the same client code reaches all of them, because the address is the only difference.

The third row is the only one where anything leaves your machine, and it never happens by accident: a request to a hosted model needs a key, and a local server with no key configured refuses to proxy anywhere.

2 · Point a client at it

llamay answers three request shapes on one port. Whichever SDK you already use, the change is the base URL and nothing else.

An OpenAI client

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:11435/v1",
    api_key="not-needed-locally",
)

answer = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarise this contract."}],
)

An Anthropic client

from anthropic import Anthropic

client = Anthropic(base_url="http://127.0.0.1:11435", api_key="local")

answer = client.messages.create(
    model="auto",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Summarise this contract."}],
)

Or nothing at all

curl http://127.0.0.1:11435/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello"}]}'

The same server also answers Ollama's /api/chat and /api/tags, so tooling written against Ollama works unmodified. See the HTTP API reference for every route.

3 · Choose a model

Name one, or send auto and let llamay pick from the question. auto is the right default for an application whose prompts vary: a one-line question does not need the model a legal summary needs, and paying for the larger one on both is how a bill becomes surprising.

llamay pull qwen2.5:7b          # by name, once
curl 127.0.0.1:11435/v1/models  # what is available right now

4 · Handle the refusals

This is the step that separates something demonstrable from something on call. llamay refuses in four ways and each one has a different correct response — retrying all of them is how a rate limit becomes an outage.

Every refusal carries a detail written to be read by the person in front of the screen. Printing it verbatim is almost always better than mapping it to a message of your own.

5 · Deploy it

There is no runtime to install and no container required. A binary, a unit file, and a directory for models.

# /etc/systemd/system/llamay.service
[Unit]
Description=llamay
After=network.target

[Service]
ExecStart=/usr/local/bin/llamay serve --addr 0.0.0.0:11435 --api-key-file /etc/llamay/key
Restart=always
User=llamay
StateDirectory=llamay

[Install]
WantedBy=multi-user.target

Bind to a loopback address unless something else needs to reach it, and set a key whenever it listens on anything other than 127.0.0.1. Serving covers the scheduler, batching and model residency; Backends covers what each kind of hardware will and will not run.

Where to go next

• Forking a context — build a prompt once and branch it, rather than resending it.
• Tools and schemas — constrain output to a JSON Schema the caller supplies.
• By industry — what changes when the data is regulated.
• HTTP API — every route, body and extension.