16 September 2026 · Guide · 4 min read

How to cut LLM latency with prompt caching

How prompt caching skips the part of a prompt a model has already read, how to see it working in llamay, and what quietly defeats it.

Reducing Latency with Prompt Caching

When running a large language model (LLM) on your own hardware, latency can be a significant concern. One effective way to reduce latency is by utilising prompt caching, a feature available in the llamay inference engine. This article explores how prompt caching works in llamay and how it can be used to improve the performance of your LLM.

To understand how prompt caching works, it is essential to know that a model call in llamay has two phases: PREFILL and DECODE. The PREFILL phase reads the prompt in one pass, while the DECODE phase writes the answer one token at a time. Prompt caching reuses the key-value state computed during the PREFILL phase, allowing repeated prefixes to be skipped. This means that if a prompt has a repeated prefix, the model will not re-read the entire prompt, resulting in reduced latency.

The following code block pulls a small chat model and starts the server, which is where the cache lives between requests:

# Pull a small chat model
llamay pull qwen2.5:0.5b

# Serve it on localhost:11435
llamay serve -m qwen2.5:0.5b

Every answer from the server then reports how many prompt tokens it did not have to read again, which is how you measure whether the cache is working (below).

Understanding the Mechanics of Prompt Caching

Prompt caching in llamay is based on a radix prefix tree over token sequences. This data structure allows for efficient reuse of computed key-value states. Additionally, llamay provides explicit control over contexts through the /v1/contexts endpoint. This endpoint allows you to create a new context, which can be forked and reused across multiple requests.

The following code block demonstrates how to create a new context and fork it:

# Create a context from a long, shared beginning
curl -X POST http://localhost:11435/v1/contexts \
  -H 'content-type: application/json' \
  -d '{"model": "qwen2.5:0.5b", "prompt": "You are the support assistant for ..."}'

# Fork it four ways, one per request
curl -X POST "http://localhost:11435/v1/contexts/<id>/fork?model=qwen2.5:0.5b&n=4"

This example creates a new context and forks it, allowing the reused prefix to be skipped in subsequent requests.

Managing Contexts and Forks

Managing contexts and forks is crucial to effectively utilising prompt caching. A fork is a page-table copy, not a memory copy, which means that it does not incur significant memory overhead. However, it is essential to note that anything that varies at the top of a prompt, such as a timestamp, can invalidate the entire cache.

The following table compares the different methods of managing contexts and forks:

In general, forking a context is the most efficient method, as it incurs minimal memory overhead.

Precision and Accuracy Tradeoffs

When using prompt caching, it is essential to consider the precision and accuracy tradeoffs. The -kv flag in llamay sets the numeric precision of the cache: f32 (the default), vq8 (values quantised, keys exact) or q8 (both quantised). This flag controls the memory and accuracy tradeoff, with higher precision resulting in increased memory usage.

The following code block demonstrates how to set the numeric precision of the cache:

# Serve with the values in the cache quantised to 8 bits, keys exact
llamay serve -m qwen2.5:0.5b -kv vq8

This keeps the keys exact and stores the values in 8 bits, which saves memory for a small loss of accuracy; q8 quantises both and saves more.

Monitoring and Optimising Prompt Caching

To monitor and optimise prompt caching, every answer from llamay serve carries usage.prompt_tokens_details.cached_tokens, the number of prompt tokens it reused instead of reading again. This metric can be used to measure the effectiveness of prompt caching and identify areas for optimisation.

The following code block demonstrates how to monitor the number of cached tokens:

# Send the same long beginning twice; the second answer reports the reuse
for i in 1 2; do
  curl -s http://localhost:11435/v1/chat/completions \
    -H 'content-type: application/json' \
    -d '{"model": "qwen2.5:0.5b", "messages": [{"role": "system", "content": "You are the support assistant for ..."}, {"role": "user", "content": "Hello"}]}' \
    | jq .usage.prompt_tokens_details.cached_tokens
done

The first request prints 0. The second prints how much of the shared beginning it skipped. If the second number is also 0, something at the top of the prompt changed between the two.

Common Pitfalls and Limitations

While prompt caching can significantly reduce latency, there are common pitfalls and limitations to be aware of. One of the most significant limitations is that anything that varies at the top of a prompt can invalidate the entire cache. This means that if a prompt includes a timestamp or other dynamic content, the cache may not be effective.

Additionally, prompt caching is not a replacement for response caching, and it is essential to understand the differences between the two. Prompt caching reuses the key-value state computed during the PREFILL phase, while response caching stores the entire response. llamay does not support response caching, and it is essential to use prompt caching effectively to reduce latency.

By understanding the mechanics of prompt caching and managing contexts and forks effectively, you can significantly reduce latency and improve the performance of your LLM. However, it is essential to be aware of the common pitfalls and limitations and to monitor and optimise prompt caching to achieve the best results.