Semantic Recall: Making Agent Memory Searchable with Embeddings

Keyword search fails when agents need to recall past experiences. If an agent stored a note about “optimizing database connection pooling,” a keyword search for “SQL performance” won’t find it — even though they’re conceptually related. Semantic recall uses embeddings to understand meaning, letting agents find relevant memories based on what they’re about, not just which words they contain.

From Keywords to Meaning

The Keyword Problem

Traditional search matches exact terms or simple patterns:

Query: "How do I make my API faster?"
Keyword Match: "fast API framework" (irrelevant)
Miss: "reduce latency by optimizing connection pools" (relevant but unmatched)

The Semantic Solution

Embeddings convert text into dense vectors where similar meanings are close together:

Query: "How do I make my API faster?"
Embedding Space: [0.23, -0.45, 0.67, ...]

Nearby Vectors:
  - "reduce API latency" (distance: 0.12) ✓
  - "optimize database connections" (distance: 0.18) ✓
  - "fast API framework" (distance: 0.45) ✗

Building Semantic Memory with Embeddings

Setting Up the Embedding Pipeline

from llama_index.core import VectorStoreIndex, Document
from llama_index.core.embeddings import resolve_embed_model
from llama_index.core.node_parser import SentenceSplitter

class SemanticMemory:
    def __init__(self, embed_model_name: str = "text-embedding-3-small"):
        self.embed_model = resolve_embed_model(f"openai/{embed_model_name}")
        self.index: Optional[VectorStoreIndex] = None
        self.memories: list[dict] = []

    def store_memory(self, content: str, metadata: dict):
        """Store a memory with semantic indexing."""
        memory = {
            "content": content,
            "metadata": metadata,
            "embedding": self.embed_model.get_text_embedding(content),
        }
        self.memories.append(memory)

        # Update vector index
        doc = Document(
            text=content,
            metadata=metadata,
            embedding=memory["embedding"]
        )
        if self.index is None:
            self.index = VectorStoreIndex.from_documents([doc])
        else:
            self.index.insert_doc(doc)

    def recall(self, query: str, top_k: int = 5) -> list[dict]:
        """Recall memories by semantic similarity."""
        if not self.index:
            return []

        query_engine = self.index.as_query_engine(
            similarity_top_k=top_k
        )
        response = query_engine.query(query)

        return [
            {
                "content": node.text,
                "score": node.score,
                "metadata": node.metadata,
            }
            for node in response.source_nodes
        ]

Choosing Embedding Models

Different embedding models have different strengths:

Model	Dimensions	Speed	Quality	Cost
text-embedding-3-small	1536	Fast	Good	Low
text-embedding-3-large	3072	Medium	Excellent	Medium
voyage-3	1024	Fast	Excellent	Medium
nomic-embed-text	768	Fast	Good	Free

For most agent memory applications, text-embedding-3-small offers the best balance of quality and cost.

Hybrid Search: Best of Both Worlds

Combine semantic search with keyword matching for maximum recall:

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.retrievers.bm25 import BM25Retriever

class HybridMemoryRecall:
    def __init__(self, index, documents):
        self.vector_retriever = VectorIndexRetriever(
            index=index,
            similarity_top_k=5,
        )
        self.bm25_retriever = BM25Retriever.from_documents(
            documents,
            similarity_top_k=5,
        )

    def recall(self, query: str, top_k: int = 5) -> list[dict]:
        """Hybrid recall combining semantic and keyword search."""
        vector_results = self.vector_retriever.retrieve(query)
        bm25_results = self.bm25_retriever.retrieve(query)

        # Reciprocal Rank Fusion
        fused = self._rrf_fusion(
            vector_results + bm25_results,
            top_k=top_k
        )
        return fused

    def _rrf_fusion(self, results, top_k, k=60):
        """Combine results using Reciprocal Rank Fusion."""
        scores = {}
        for rank, result in enumerate(results):
            doc_id = result.node.id_
            if doc_id not in scores:
                scores[doc_id] = {
                    "result": result,
                    "score": 0,
                }
            scores[doc_id]["score"] += 1 / (k + rank + 1)

        sorted_results = sorted(
            scores.values(),
            key=lambda x: x["score"],
            reverse=True
        )

        return [
            {
                "content": r["result"].node.text,
                "score": r["score"],
                "metadata": r["result"].node.metadata,
            }
            for r in sorted_results[:top_k]
        ]

Memory Indexing Strategies

Chunk-Level Indexing

Split long memories into chunks for more precise retrieval:

splitter = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=50,
)

documents = splitter.splitDocuments(
    [Document(text=long_memory)]
)

Hierarchical Indexing

Create a two-level index: summaries for broad search, details for precise recall:

# Level 1: Summary embeddings (fast, broad)
summary_index = VectorStoreIndex.from_documents(summary_docs)

# Level 2: Detail embeddings (precise, slower)
detail_index = VectorStoreIndex.from_documents(detail_docs)

def hierarchical_recall(query: str):
    # First, find relevant summaries
    summaries = summary_index.as_query_engine().query(query)

    # Then, search within matched summary regions
    for summary in summaries.source_nodes:
        details = detail_index.as_query_engine().query(
            f"Details about: {summary.text[:100]}"
        )
        yield from details.source_nodes

Temporal Indexing

Weight recent memories higher while still allowing older memories:

def temporal_boost(score: float, created_at: datetime, decay_rate: float = 0.01):
    """Boost score based on recency."""
    days_old = (datetime.now() - created_at).days
    recency_factor = math.exp(-decay_rate * days_old)
    return score * (1 + recency_factor * 0.3)

Best Practices

1. Use consistent embedding models — Once you index memories with a specific model, use the same model for queries. Mixing models produces incompatible vectors.

2. Store raw text alongside embeddings — Embeddings are for search; you still need the original text for context. Always store the full content with each indexed memory.

3. Batch embedding calls — When indexing many memories at once, batch them to reduce API calls and improve throughput.

4. Implement feedback loops — Track which recalled memories are actually useful. Use this signal to improve retrieval over time.

5. Cache embeddings — Don’t re-embed unchanged memories. Use a hash of the content as a cache key.

Conclusion

Semantic recall transforms agent memory from a keyword-matching system into a meaning-understanding system. By using embeddings to index memories, agents can find relevant past experiences even when the query words don’t match the stored content exactly. Combined with hybrid search strategies, this creates a memory system that’s both flexible and precise.

References:

• Hugging Face Embeddings Documentation
• LlamaIndex Vector Store Index