Prompt Caching for Agentic Loops: Cutting Costs in Multi-Turn Systems

Multi-turn agent systems are expensive. Every tool call, every conversation turn, every re-evaluation of system instructions incurs token costs. Prompt caching is the practice of storing frequently used context so it doesn’t need to be reprocessed on every API call.

In agentic loops—where the model makes multiple sequential decisions—prompt caching can reduce costs by 50-90% while maintaining the same quality of output.

The Cost Problem in Agentic Systems

Consider a typical agentic workflow:

1. System instructions (~500 tokens)
2. Tool definitions (~300 tokens)
3. Conversation history (~2,000 tokens)
4. Current turn input (~200 tokens)

Total: ~3,000 tokens per turn

In a 10-turn agent loop, that’s 30,000 input tokens—most of which are repeated across turns. Without caching, you pay for reprocessing the same context repeatedly.

How Prompt Caching Works

Prompt caching stores processed context on the server side. When the same prefix is sent in a subsequent request, the provider skips reprocessing and applies cached results.

Provider Support

Provider	Caching Type	Savings
OpenAI	Automatic prefix caching	Up to 50% on cached tokens
Google Gemini	Context caching with TTL	Up to 75% on cached context
Anthropic	Prompt caching	Up to 90% on cached prefixes

The Cache Key

The cache key is the exact prefix of your prompt. Any change to the beginning invalidates the cache:

✅ Cached: [system_instructions][tool_definitions][history][new_input]
✅ Cached: [system_instructions][tool_definitions][history][new_input]
❌ Not cached: [modified_system][tool_definitions][history][new_input]

Caching Strategies for Agents

Strategy 1: Stable Prefix

Keep system instructions and tool definitions identical across all requests:

# These should NEVER change between turns
SYSTEM_INSTRUCTIONS = "You are a helpful assistant..."
TOOL_DEFINITIONS = [...]  # Same tools for every turn

def build_prompt(history, current_input):
    return {
        "system": SYSTEM_INSTRUCTIONS,  # Always the same
        "tools": TOOL_DEFINITIONS,       # Always the same
        "messages": history + [{"role": "user", "content": current_input}],
    }

Strategy 2: Conversation History Ordering

Structure conversation history to maximize cache hits:

# Good: History grows from the left, new content on the right
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "Hello"},      # Cached from turn 1
    {"role": "assistant", "content": "Hi!"},    # Cached from turn 1
    {"role": "user", "content": "Help me"},    # Cached from turn 2
    {"role": "assistant", "content": "Sure"},   # Cached from turn 2
    {"role": "user", "content": "New question"}, # Only this is new
]

# Bad: Inserting messages at the beginning invalidates cache
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "New question"},  # Changed the prefix!
    {"role": "assistant", "content": "Sure"},
    {"role": "user", "content": "Help me"},
]

Strategy 3: Context Windowing

When history grows too long, summarize older turns but keep the prefix stable:

def window_history(history, max_turns=10):
    if len(history) <= max_turns:
        return history

    # Keep the last N turns in full
    recent = history[-max_turns:]

    # Summarize older turns into a single message
    older = history[:-max_turns]
    summary = summarize(older)

    return [
        {"role": "system", "content": f"Previous context: {summary}"},
        *recent,
    ]

Strategy 4: Tool Result Caching

Cache tool results to avoid re-executing expensive operations:

from functools import lru_cache

@lru_cache(maxsize=100)
def expensive_tool(query: str) -> str:
    # This result is cached
    return run_expensive_query(query)

Gemini Context Caching

Google Gemini provides explicit context caching with TTL:

import google.generativeai as genai

# Create a cache for frequently used context
cache = genai.create_cached_content(
    name="agent-context-cache",
    model="gemini-2.0-flash",
    system_instruction="You are a helpful assistant.",
    tools=[...],
    contents=[...],
    ttl=datetime.timedelta(hours=1),
)

# Use the cached context in requests
response = model.generate_content(
    "New question here",
    cached_content=cache.name,
)

Cache Pricing

Gemini charges for cache creation and storage:

Operation	Cost
Cache creation	$1.25 / 1M tokens
Cache storage	$0.00625 / 1M tokens / hour
Cache read	25% of base input price

For a 10,000-token system instruction cached for 1 hour:

• Creation: $0.0125
• Storage: $0.0000625
• Total: ~$0.013 vs $0.05 without caching (74% savings)

Measuring Cache Effectiveness

Track these metrics:

# Cache hit rate
cache_hits / total_requests

# Cost savings
(cost_without_cache - cost_with_cache) / cost_without_cache

# Latency improvement
(avg_latency_without_cache - avg_latency_with_cache) / avg_latency_without_cache

Implementation Pattern

Here’s a complete pattern for adding prompt caching to an agent loop:

import hashlib
from typing import Dict, Any

class PromptCache:
    def __init__(self):
        self.cache: Dict[str, Any] = {}

    def get_cache_key(self, messages):
        """Generate a cache key from the message prefix."""
        prefix = str(messages[:3])  # System + first 2 turns
        return hashlib.md5(prefix.encode()).hexdigest()

    def build_prompt(self, system, tools, history, new_input):
        """Build a prompt with stable prefix for caching."""
        messages = [
            {"role": "system", "content": system},
            *history,
            {"role": "user", "content": new_input},
        ]
        return messages

# Usage
cache = PromptCache()

for turn in range(max_turns):
    messages = cache.build_prompt(
        SYSTEM_INSTRUCTIONS,
        TOOL_DEFINITIONS,
        conversation_history,
        user_input,
    )

    response = await llm.generate(messages)

    # Update history (grows from the right)
    conversation_history.append({"role": "user", "content": user_input})
    conversation_history.append({"role": "assistant", "content": response})

Common Pitfalls

1. Changing the prefix: Any modification to the beginning of the prompt invalidates the cache
2. Dynamic tool definitions: If tools change per turn, they can’t be cached
3. Timestamp injection: Adding timestamps to system instructions breaks caching
4. Cache staleness: Long-running sessions may use outdated cached context

Best Practices

1. Separate static and dynamic content: Keep instructions and tools separate from dynamic data
2. Append, don’t prepend: Add new content to the end of messages
3. Monitor cache hit rates: Track how often your cache is actually used
4. Set appropriate TTLs: Balance cache freshness with cost savings
5. Test with and without: Verify that caching doesn’t degrade quality

Next Steps

• Explore Semantic Caching for fuzzy cache matching
• Read about Cost Monitoring to track your savings
• Learn about Context Engineering for broader context management

Prompt caching is one of the highest-ROI optimizations you can make in agentic systems. By keeping the prefix stable and appending new content, you can cut costs dramatically while maintaining the same agent performance.