Prompt caching
Prompt caching lets an AI provider reuse the work it already did on the start of a prompt when the next request begins with exactly the same text. Instead of processing a long system prompt, a set of documents or a conversation history from scratch on every call, the model picks up from a saved state and only processes what’s new.
Why it matters
Agents and chat apps send the same material again and again. Every turn of a conversation resends the whole history, and every step of an agent resends its instructions, tool definitions and the files it has read. Without caching, you pay full price to process that repeated text each time.
With caching, providers charge much less for the cached part, and responses start sooner because the model skips reprocessing it. In long conversations and agent sessions, repeated input is often most of what gets sent, so caching can make a large difference to what a workload costs.
Using it well
- Stable content first, changing content last. The cache matches from the beginning of the prompt, and the first difference ends the match. Put system instructions, tool definitions and reference documents at the top, and the new message at the end.
- Leave the top alone. A timestamp, a random ID or a reshuffled list of tools near the start of the prompt breaks the cache for everything after it.
- Mind the timer. Caches expire after a stretch without use, often a few minutes, so work that pauses for longer pays full price on the next call.
- Check the numbers. Most providers report cached and uncached tokens separately. If the cached share is low, something near the top of your prompt keeps changing.
Some providers cache automatically. Others need you to mark which part of the prompt to cache, so check how yours works.