Prompt Caching

Prompt caching reuses the system prompt, tools, and earlier messages that each request sends again. The provider bills those tokens once while the cache is valid.

chat() turns it on by default. A cache hit also makes the answer start sooner.

Quick setup#

This setup takes about 2 minutes.

1. Pass the threadId to chat()#

chat() uses the threadId as the cache key:

import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

export async function POST(request: Request) {
  const { messages, threadId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: openaiText("gpt-6.1-sol"),
    systemPrompts: ["You are a support agent for Acme."],
    messages,
    threadId,
  });

  return toServerSentEventsResponse(stream);
}

The client does not change. useChat already sends one threadId for each conversation.

2. See the cache work#

Log the cached tokens in the onUsage hook of a middleware. The hook runs one time for each model call:

import { chat } from "@tanstack/ai";
import type { ChatMiddleware } from "@tanstack/ai";
import { anthropicText } from "@tanstack/ai-anthropic";

const cacheLog: ChatMiddleware = {
  name: "cache-log",
  onUsage: (ctx, usage) => {
    const read = usage.promptTokensDetails?.cachedTokens ?? 0;
    console.log(`Call ${ctx.iteration}: ${read} of ${usage.promptTokens} tokens from the cache`);
  },
};

const stream = chat({
  adapter: anthropicText("claude-sonnet-5-5"),
  messages: [{ role: "user", content: "Hello!" }],
  threadId: "thread-1",
  middleware: [cacheLog],
});

From the second call of a conversation, most of the input comes from the cache.

How it works#

• The cache key is the first of these that you set:
  1. promptCache.key.
  2. The threadId that you pass to chat().
  3. No key. Claude does not need one. OpenAI uses the key to route requests, so a stable threadId gives more cache hits.
• A request reads the cache only up to the first token that is different. Text that changes on each call, such as the current time in the system prompt, ends the match there.
• usage.promptTokens is the full input. Two fields split it:
  • usage.promptTokensDetails.cachedTokens: tokens read from the cache.
  • usage.promptTokensDetails.cacheWriteTokens: tokens written to the cache. Claude reports this field.

Advanced#

Turn it off, or keep it longer#

Set promptCache on chat():

To set your own key too, pass an object. Both fields are optional:

import { chat } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

const stream = chat({
  adapter: openaiText("gpt-6.1-sol"),
  messages: [{ role: "user", content: "Where is my order?" }],
  threadId: "thread-1",
  promptCache: { retention: "long", key: "acme-support" },
});

The types come from @tanstack/ai: PromptCacheRetention, PromptCacheOptions, and ResolvedPromptCache.

To change the cache for one model call only, use a middleware. See Change the prompt cache of a call.

Know what it costs on Claude#

On OpenAI and Mistral, a cache write is free and a cache read costs less. Keep the default.

Claude bills the cache write, and a read costs much less than normal input:

• With the 5-minute cache ('short'), a write costs a bit more than normal input. A start that you send two times already costs less than no cache.
• With the 1-hour cache ('long'), a write costs about two times the normal input. The start must be read more times before the cache saves money.

Two cases change the choice:

• A large prompt that you send one time. For example, a long document in each call that you never send again. Nothing reads that cache, so the write is only extra cost. Use 'none'.
• Long pauses. The short Claude cache ends 5 minutes after the last read. If your users often reply later than that, use 'long'.

import { chat } from "@tanstack/ai";
import { anthropicText } from "@tanstack/ai-anthropic";

export function reviewContract(contract: string) {
  return chat({
    adapter: anthropicText("claude-sonnet-5-5"),
    messages: [{ role: "user", content: `Review this contract:\n\n${contract}` }],
    threadId: "thread-1",
    // Sent one time only: skip the cache write.
    promptCache: "none",
  });
}

Claude caches a start only when it is longer than a minimum for the model (512 to 4096 tokens). A shorter start costs nothing extra.

What each provider gets#

Other adapters send no cache fields.

Each adapter page has the details: Anthropic, Amazon Bedrock, OpenAI, OpenRouter, and Mistral.

Set the cache yourself#

Your own cache fields win. Then chat() adds no automatic fields of that kind to the request:

• Anthropic: cache_control in the metadata of a system prompt, a message part, or a tool, or modelOptions.cache_control.
• Bedrock: metadata.cachePoint on a system prompt, a message part, or a tool.
• OpenAI: modelOptions.prompt_cache_key or modelOptions.prompt_cache_retention.
• OpenRouter: modelOptions.sessionId.

Add tools without losing the cache#

The tools sit near the start of each request. If you add a tool during a run, the next call cannot read the cache from that point. Some GPT and Claude models take the new tool inside the conversation, so the start stays the same. See Mid-Conversation Changes.