Engineering

Building and releasing

Go 1.26, no cgo on the default build, and nothing outside the standard library. That is what makes six targets buildable from one machine — and it is also the property every GPU backend was designed around rather than allowed to break.

Building

make build          # ./bin/llamay — portable, no cgo
make build-metal    # + Metal
make build-fast     # + Metal and Accelerate
make build-cuda     # + CUDA
make build-vulkan   # + Vulkan
make build-gpu      # CUDA and Vulkan in one binary, chosen at run time
make server         # also links bin/llamay-server to the same binary
make cross          # every shipped target from one machine
make cross-portable # riscv64, s390x, ppc64le and wasm

build-cuda needs a C compiler and nothing else — no CUDA toolkit, no headers, no card. The driver API is resolved with dlopen at first use rather than linked, so the binary it produces runs on a machine with an NVIDIA driver and builds on one without. The toolkit is needed only to regenerate the PTX, which is checked in. Vulkan is the same shape with the ABI transcribed instead.

Testing

make test           # every package
make check          # fmt, vet, test, and verify against a generated model
LLAMAY_BACKEND=portable make test   # the reference kernels, on a machine with SIMD

CI is 14 jobs: three operating systems, five backend tag sets, the race detector, every architecture shape, and 12 cross-compile targets. It asserts with ldd on every build that a GPU binary links no driver, because that is the kind of property which regresses the moment somebody adds a convenient import.

How a release is made

Why the install scripts are served from here

curl https://llamay.com/install.sh | sh is a promise that the thing served is the thing in the repository. So the deploy copies packaging/install.sh and packaging/install.ps1 to the site root on every push that touches them — deploying from the repository is the only way to keep the reviewed script and the executed one from drifting.

Both are parse-checked before they are served rather than after somebody reports that the pipe did nothing: sh -n, then dash -n as well, because the one-liner is piped into /bin/sh, which on Debian and Ubuntu is dash and where a bashism would only show up there. The PowerShell script goes through the language parser.

These pages

The docs are generated by scripts/site/build_docs.py and the output is committed. Cloudflare Pages serves site/ as static files with no build step, and adding one would mean the thing reviewed and the thing served are produced by different processes. The deploy instead re-runs the generator with --check and fails if the tree changes, so the committed HTML is provably what the generator emits.

python3 scripts/site/build_docs.py           # write site/docs/
python3 scripts/site/build_docs.py --check   # what CI runs

Repository layout

cmd/llamay          the CLI
pkg/store           the model store: content-addressed blobs and manifests
pkg/ollama          reading a local Ollama store, if there is one
pkg/gguf            format: parse, mmap, write
pkg/quant           block layouts, dequantisation, fused W4A8 kernels
pkg/tensor          matrices, matvec for decode, matmul for prefill, worker pool
pkg/tok             byte-level BPE, SentencePiece and WordPiece
pkg/arch            supported architectures and their configuration
pkg/model           weight binding, forward pass, batched prefill
pkg/kv              the state engine: paging, prefix tree, fork, snapshots
pkg/sample          samplers and the constraint interface
pkg/constrain       JSON, JSON Schema, GBNF and lexicon constraints
pkg/ocr             CTC decoding with lexicon beam search
pkg/serve           OpenAI, Anthropic, Ollama and context HTTP APIs
site/               llamay.com — this site, deployed from here
scripts/site/       the generator for site/docs/