Vector Memory for Agents: Embedding and Indexing Conversation History

Agents forget. A conversation from yesterday that contained critical context? Gone. A user preference established five turns ago? Lost in the context window. Vector memory solves this by embedding every conversation turn and indexing it for semantic retrieval, giving agents true long-term recall.

The Problem with Rolling Context

Standard LLM applications keep a sliding window of recent messages. This works for short conversations but fails when:

• Users reference topics from hours or days ago
• Agents need to maintain preferences across sessions
• Complex workflows span multiple interactions
• The conversation exceeds the model’s context window

Vector memory treats every conversation turn as a searchable document. When the agent needs context, it retrieves the most relevant past interactions — not just the most recent ones.

Architecture Overview

A vector memory system has four components:

Conversation Turn
       ↓
   Embedding Model (e.g., text-embedding-3-small)
       ↓
   Vector Store (e.g., Qdrant, Pinecone, pgvector)
       ↓
   Semantic Search → Top-K Relevant Turns
       ↓
   Inject into Agent Context

Implementation

Setting Up the Vector Store

from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct
from openai import OpenAI
import uuid
from datetime import datetime

class VectorMemory:
    def __init__(self, collection_name: str = "agent_conversations"):
        self.client = QdrantClient(":memory:")  # Use remote for production
        self.embedder = OpenAI()
        self.collection_name = collection_name
        self._ensure_collection()
    
    def _ensure_collection(self):
        collections = self.client.get_collections().collections
        if not any(c.name == self.collection_name for c in collections):
            self.client.create_collection(
                collection_name=self.collection_name,
                vectors_config=VectorParams(
                    size=1536,  # text-embedding-3-small dimension
                    distance=Distance.COSINE
                )
            )
    
    def _embed(self, text: str) -> list[float]:
        response = self.embedder.embeddings.create(
            model="text-embedding-3-small",
            input=text
        )
        return response.data[0].embedding

Storing Conversation Turns

Each conversation turn gets embedded and stored with rich metadata:

def store_turn(
    self,
    thread_id: str,
    role: str,
    content: str,
    turn_id: int,
    metadata: dict = None
):
    embedding = self._embed(content)
    
    point = PointStruct(
        id=str(uuid.uuid4()),
        vector=embedding,
        payload={
            "thread_id": thread_id,
            "role": role,
            "content": content,
            "turn_id": turn_id,
            "timestamp": datetime.now().isoformat(),
            "metadata": metadata or {}
        }
    )
    
    self.client.upsert(
        collection_name=self.collection_name,
        points=[point]
    )

Semantic Retrieval

When the agent needs context, retrieve the most relevant past turns:

def retrieve_context(
    self,
    query: str,
    thread_id: str = None,
    top_k: int = 5,
    min_score: float = 0.7
) -> list[dict]:
    query_embedding = self._embed(query)
    
    # Build filter for specific thread
    filter_condition = None
    if thread_id:
        from qdrant_client.models import Filter, FieldCondition, MatchValue
        filter_condition = Filter(
            must=[
                FieldCondition(
                    key="thread_id",
                    match=MatchValue(value=thread_id)
                )
            ]
        )
    
    results = self.client.search(
        collection_name=self.collection_name,
        query_vector=query_embedding,
        limit=top_k,
        score_threshold=min_score,
        query_filter=filter_condition
    )
    
    return [
        {
            "content": hit.payload["content"],
            "role": hit.payload["role"],
            "score": hit.score,
            "timestamp": hit.payload["timestamp"]
        }
        for hit in results
    ]

Integrating with an Agent

The vector memory integrates seamlessly into an agent’s workflow:

from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

def create_memory_augmented_agent(vector_memory: VectorMemory):
    prompt = ChatPromptTemplate.from_messages([
        ("system", """You are a helpful assistant with access to conversation history.
        
When relevant to the user's question, retrieved context from past conversations will be
provided below. Use this context to provide personalized, continuity-aware responses."""),
        MessagesPlaceholder(variable_name="retrieved_context"),
        MessagesPlaceholder(variable_name="chat_history"),
        ("human", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ])
    
    agent = create_openai_tools_agent(llm, tools, prompt)
    
    return AgentExecutor(
        agent=agent,
        tools=tools,
        memory=vector_memory,
        verbose=True
    )

Advanced Techniques

Conversation Chunking

Instead of storing individual turns, chunk conversations into meaningful segments:

def chunk_conversation(messages: list[dict], chunk_size: int = 5) -> list[dict]:
    chunks = []
    for i in range(0, len(messages), chunk_size):
        chunk = messages[i:i + chunk_size]
        combined_text = "\n".join(
            f"{m['role']}: {m['content']}" for m in chunk
        )
        chunks.append({
            "content": combined_text,
            "start_turn": i,
            "end_turn": min(i + chunk_size, len(messages))
        })
    return chunks

Hybrid Search

Combine vector similarity with keyword matching for better retrieval:

def hybrid_search(
    self,
    query: str,
    thread_id: str = None,
    top_k: int = 5
) -> list[dict]:
    # Vector search
    vector_results = self.retrieve_context(query, thread_id, top_k * 2)
    
    # Keyword search (BM25 or similar)
    keyword_results = self.keyword_search(query, thread_id, top_k * 2)
    
    # Reciprocal Rank Fusion
    fused = reciprocal_rank_fusion([vector_results, keyword_results])
    return fused[:top_k]

Temporal Weighting

Weight recent conversations higher while still allowing older relevant matches:

import math

def temporal_boost(score: float, timestamp: str, half_life_days: int = 7) -> float:
    age_days = (datetime.now() - datetime.fromisoformat(timestamp)).days
    decay = math.exp(-0.693 * age_days / half_life_days)
    return score * (0.7 + 0.3 * decay)  # 70% relevance, 30% recency

Storage Considerations

Approach	Latency	Cost	Best For
In-memory (Qdrant :memory:)	<1ms	Free	Development
Local SQLite + vectors	1-5ms	Free	Single-user apps
Qdrant Cloud	5-20ms	$$	Production SaaS
Pinecone	10-30ms	$$$	Enterprise scale
pgvector	5-15ms	$	PostgreSQL shops

Privacy and Retention

Vector memory raises important privacy considerations:

• Data retention policies — Automatically delete old embeddings
• User deletion requests — Implement GDPR-compliant data removal
• Consent management — Track what data users agreed to store
• Encryption at rest — Encrypt vector stores containing sensitive conversations

def enforce_retention(self, max_age_days: int = 90):
    cutoff = (datetime.now() - timedelta(days=max_age_days)).isoformat()
    self.client.delete(
        collection_name=self.collection_name,
        points_selector=Filter(
            must=[
                FieldCondition(
                    key="timestamp",
                    range=Range(lt=cutoff)
                )
            ]
        )
    )

Conclusion

Vector memory transforms agents from stateless responders into persistent collaborators. By embedding and indexing every conversation turn, agents can recall relevant context from any point in their history — not just the recent past. Start with a simple in-memory vector store, add temporal weighting for natural decay, and graduate to a managed vector database as your agent scales.