Semantic Caching to Reduce Latency and Spend

“What is your return policy?” and “How do I return a product?” are three different strings and one semantic intent. A traditional cache — which matches exact text — misses all of them, and each one bills a fresh LLM call. A semantic cache embeds each query into a vector, compares it by meaning against previously cached queries, and serves the stored response when similarity clears a threshold. No LLM call needed.

Measured in production-style workloads, the payoff is large: one AWS evaluation of 63,796 real chatbot queries found semantic caching cut LLM inference cost by up to 86% and average latency by 88%, at a threshold of 0.75, while holding ~91% answer accuracy.

Prerequisites

• Python 3.10+
• Redis (local or cloud) and pip install redis redisvl
• An embedding model (the examples use a text embedder; local BGE-small avoids a network hop)

How a semantic cache works

1. Embed the incoming query into a vector.
2. Search the cache’s vector index for stored query embeddings within a similarity threshold.
3. Hit: return the cached response immediately — only an embedding call, milliseconds, no LLM.
4. Miss: call the LLM, then store the query’s embedding + response for future reuse.

from redisvl.extensions.llmcache import SemanticCache
from redisvl.utils.vectorize import HFTextVectorizer

cache = SemanticCache(
    name="llmcache",
    redis_url="redis://localhost:6379",
    distance_threshold=0.1,                        # semantic distance (lower = stricter)
    vectorizer=HFTextVectorizer("sentence-transformers/all-MiniLM-L6-v2"),
)

Wiring the cache into your LLM call

def answer(question: str) -> str:
    if results := cache.check(prompt=question):
        return results[0]["response"]              # cache hit — no LLM call

    response = ask_llm(question)                   # cache miss — the expensive path
    cache.store(prompt=question, response=response, metadata={"model": "gemini-2.5-pro"})
    return response

On a miss you pay the embedding + vector lookup (5–20ms) plus normal LLM latency. On a hit you pay only the lookup and return in milliseconds — the RedisVL docs show a ~97% time saving on a representative question compared to a fresh LLM call.

Tuning the threshold

The similarity threshold is the single critical knob — it trades savings against quality:

• Too strict (0.99): few hits, little savings.
• Too relaxed (0.50): high hit rate but wrong answers for genuinely different queries.
• Sweet spot for most apps: 0.75–0.95 cosine similarity, depending on domain.

From the AWS benchmark on real traffic:

Threshold	Hit ratio	Accuracy	Cost saved	Latency (s)
None	—	—	—	4.35
0.95	56%	92.6%	51.9%	1.84
0.90	74.5%	92.3%	72.5%	1.21
0.80	87.6%	91.8%	84.6%	0.60
0.75	90.3%	91.2%	86.3%	0.51

TTL and invalidation

Cached answers go stale when your knowledge changes. Set a TTL per entry — short for chat-like content, longer for FAQ corpora — and bump a corpus version in the cache key when data updates, invalidating old entries lazily as their TTL expires.

cache = SemanticCache(..., ttl=86400)   # expire entries after 1 day

Multi-tenant isolation

A cache must never leak answers across tenants. Tag entries with the tenant and filter on it at lookup time — otherwise a user’s answer (or another customer’s data) can surface in a semantically similar query from a different tenant.

Where it pays (and where it doesn’t)

• Customer support / knowledge-base assistants: 40–65% hit rates after the cache warms — the best case. Repeated question patterns make the cache pay for itself within days.
• Internal docs assistants: 30–50% hit rate.
• Open-ended creative/analytical apps: 5–15% — the embedding overhead can exceed the savings. Don’t semantic-cache queries that are almost always novel.

Below roughly a few hundred similar queries per day, the embedding + index overhead may not be worth it — exact-match or prompt caching is enough.

Putting It All Together

A FastAPI endpoint with a semantic cache in front of the LLM:

from fastapi import FastAPI, Request

app = FastAPI()

@app.post("/answer")
async def answer(request: Request):
    q = (await request.json())["question"]
    if results := cache.check(prompt=q, filters={"tenant": current_tenant}):
        return {"answer": results[0]["response"], "cached": True}
    resp = ask_llm(q)
    cache.store(prompt=q, response=resp,
                metadata={"tenant": current_tenant})
    return {"answer": resp, "cached": False}

Conclusion & Next Steps

Semantic caching turns your most-repeated questions into millisecond responses and cuts spend by roughly half to 86% depending on query redundancy. Next: tune the threshold against a small eval set before going live, add tenant-scoped filters, wire hit rate as an SLI, and pre-seed the cache with your FAQ library so it starts warm instead of warming for a week.

References / Sources

• RedisVL LLM caching. https://redis.io/docs/latest/develop/ai/redisvl/user_guide/llmcache/
• Lower cost and latency using ElastiCache as a semantic cache with Bedrock. https://aws.amazon.com/blogs/database/lower-cost-and-latency-for-ai-using-amazon-elasticache-as-a-semantic-cache-with-amazon-bedrock/
• What is semantic caching? https://redis.io/blog/what-is-semantic-caching/
• Reducing LLM costs and latency via semantic embedding caching. https://arxiv.org/abs/2411.05276