Agent Memory: Vector Stores, Knowledge Graphs, and Long-Term Recall

An AI agent that forgets everything between conversations is not an agent — it is a chatbot with extra steps. The difference between a useful assistant and a forgetful one comes down to memory architecture: how the agent stores, retrieves, and reasons over past interactions, learned facts, and evolving context.

In this post, we explore three approaches to agent memory — vector stores, knowledge graphs, and hybrid systems — and show how to implement each with LlamaIndex.

Why This Matters

LLMs have a fixed context window. Even with 128K or 1M token models, you cannot stuff an entire conversation history, a user profile, and a knowledge base into every prompt. Memory systems solve this by providing selective recall — pulling the most relevant past information into the current context.

For agents specifically, memory is not optional. An agent that cannot remember what it learned five minutes ago will repeat mistakes, ask redundant questions, and fail at multi-step tasks. Long-term memory transforms agents from stateless tools into persistent collaborators.

The key tradeoffs are:

• Vector stores excel at semantic similarity search — “find me the most relevant past conversation about X”
• Knowledge graphs excel at structured reasoning — “what does the user prefer, and how does that relate to their current request?”
• Hybrid systems combine both for the best of each world

Approach 1: Vector Store Memory

Vector stores are the simplest memory layer for agents. Every interaction gets embedded and stored; retrieval uses cosine similarity to find the most relevant past memories.

Setting Up a Vector Memory Store

With LlamaIndex, you can build a vector memory system in a few lines:

from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.qdrant import QdrantVectorStore

Settings.llm = OpenAI(model="gpt-4o")

# Create a persistent vector store for memory
vector_store = QdrantVectorStore(
    collection_name="agent_memory",
    url="http://localhost:6333",
    api_key="your-api-key",
)

# Build a memory index from past interactions
memory_index = VectorStoreIndex.from_vector_store(vector_store)

Storing Interactions as Memories

Each interaction becomes a memory node with metadata:

from llama_index.core.schema import TextNode
from datetime import datetime

def store_memory(memory_index, user_id: str, role: str, content: str, context: str = ""):
    """Store an interaction as a memory node."""
    node = TextNode(
        text=content,
        metadata={
            "user_id": user_id,
            "role": role,
            "timestamp": datetime.now().isoformat(),
            "context": context,
        },
    )
    memory_index.insert_nodes([node])
    memory_index.vector_store.persist()

Retrieving Relevant Memories

Before responding, query the memory store for relevant past context:

def recall_memories(memory_index, query: str, user_id: str, top_k: int = 5):
    """Retrieve the most relevant memories for a query."""
    query_engine = memory_index.as_query_engine(
        similarity_top_k=top_k,
        filters={"user_id": user_id},
    )
    response = query_engine.query(query)
    return response.response

The Problem With Pure Vector Memory

Vector similarity is powerful but has a fundamental weakness: it finds memories that sound similar, not memories that are logically relevant. If a user says “What was that restaurant I mentioned last Tuesday?”, vector similarity might return memories about other restaurants rather than the specific one from last Tuesday.

This is where knowledge graphs become essential.

Approach 2: Knowledge Graph Memory

Knowledge graphs store information as entities and relationships rather than text embeddings. This enables structured queries like “What restaurants does the user like?” or “What is the relationship between Entity A and Entity B?”

Building a Knowledge Graph With LlamaIndex

LlamaIndex provides a KnowledgeGraphIndex that automatically extracts entities and relationships:

from llama_index.core import KnowledgeGraphIndex
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.graph_stores.neo4j import Neo4jGraphStore

# Set up a Neo4j-backed knowledge graph
graph_store = Neo4jGraphStore(
    url="bolt://localhost:7687",
    username="neo4j",
    password="your-password",
)

# Create a knowledge graph index
from llama_index.core import StorageContext

storage_context = StorageContext.from_defaults(graph_store=graph_store)

kg_index = KnowledgeGraphIndex.from_documents(
    documents=documents,
    storage_context=storage_context,
    max_triplets_per_chunk=5,
    include_embeddings=True,
)

Extracting Structured Facts

Knowledge graphs automatically extract entity-relation-entity triples:

# Example extracted triples from conversation:
# (User, prefers, Italian food)
# (Italian restaurant, located_in, downtown)
# (User, visited, Mario's on Tuesday)
# (Mario's, serves, Italian food)

Querying the Knowledge Graph

Natural language queries are translated into graph traversals:

query_engine = kg_index.as_query_engine(
    include_text=True,
    response_mode="tree_summarize",
)

# This query leverages graph structure
response = query_engine.query(
    "What type of food does the user prefer and what restaurants match?"
)

The knowledge graph回答s this by traversing relationships: User → prefers → Italian food → served_by → Italian restaurants → located_in → downtown.

Approach 3: Hybrid Memory Architecture

The most effective agent memory systems combine vector stores for semantic recall with knowledge graphs for structured reasoning. Here is a practical hybrid implementation:

from llama_index.core import VectorStoreIndex, KnowledgeGraphIndex
from llama_index.core.schema import QueryBundle

class HybridMemory:
    def __init__(self, user_id: str):
        self.user_id = user_id
        self.vector_store = VectorStoreIndex(...)
        self.knowledge_graph = KnowledgeGraphIndex(...)

    def store_interaction(self, content: str, context: str = ""):
        """Store in both vector store and knowledge graph."""
        # Vector store for semantic recall
        self.vector_store.insert_nodes([TextNode(
            text=content,
            metadata={"user_id": self.user_id, "context": context},
        )])

        # Knowledge graph for structured facts
        self.knowledge_graph.insert_nodes([TextNode(
            text=content,
            metadata={"user_id": self.user_id},
        )])

    def recall(self, query: str, mode: str = "hybrid") -> str:
        """Recall memories using the specified mode."""
        if mode == "semantic":
            return self._semantic_recall(query)
        elif mode == "structured":
            return self._structured_recall(query)
        else:
            # Combine both approaches
            semantic_results = self._semantic_recall(query, top_k=3)
            structured_results = self._structured_recall(query, top_k=3)
            return self._merge_results(semantic_results, structured_results)

    def _semantic_recall(self, query: str, top_k: int = 5) -> list:
        engine = self.vector_store.as_query_engine(similarity_top_k=top_k)
        return engine.query(query).response

    def _structured_recall(self, query: str, top_k: int = 5) -> list:
        engine = self.knowledge_graph.as_query_engine(
            include_text=True, response_mode="tree_summarize"
        )
        return engine.query(query).response

    def _merge_results(self, semantic: list, structured: list) -> str:
        """Merge and deduplicate results from both systems."""
        # Use reranking to combine results
        all_results = semantic + structured
        # Deduplicate by content hash, keep highest relevance
        seen = set()
        merged = []
        for result in all_results:
            content_hash = hash(result)
            if content_hash not in seen:
                seen.add(content_hash)
                merged.append(result)
        return "\n".join(merged[:5])

Getting Started Tutorial

Here is a step-by-step guide to building an agent with persistent memory:

Step 1: Install Dependencies

pip install llama-index-core llama-index-llms-openai \
    llama-index-vector-stores-qdrant llama-index-graph-stores-neo4j

Step 2: Set Up Infrastructure

Start Qdrant (vector store) and Neo4j (knowledge graph):

# Qdrant
docker run -p 6333:6333 qdrant/qdrant

# Neo4j
docker run -p 7687:7687 -e NEO4J_AUTH=neo4j/password neo4j:latest

Step 3: Create the Memory System

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.graph_stores.neo4j import Neo4jGraphStore
from llama_index.core import VectorStoreIndex, KnowledgeGraphIndex, StorageContext

Settings.llm = OpenAI(model="gpt-4o")

# Initialize stores
vector_store = QdrantVectorStore(collection_name="memory", url="http://localhost:6333")
graph_store = Neo4jGraphStore(url="bolt://localhost:7687", username="neo4j", password="password")

vector_index = VectorStoreIndex.from_vector_store(vector_store)
kg_index = KnowledgeGraphIndex.from_vector_store(vector_store, storage_context=StorageContext.from_defaults(graph_store=graph_store))

Step 4: Build a Memory-Aware Agent

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import QueryEngineTool

# Create tools from memory indices
memory_tool = QueryEngineTool.from_defaults(
    query_engine=vector_index.as_query_engine(),
    name="semantic_memory",
    description="Retrieve memories based on semantic similarity",
)

kg_tool = QueryEngineTool.from_defaults(
    query_engine=kg_index.as_query_engine(),
    name="knowledge_memory",
    description="Query structured knowledge about users and relationships",
)

# Create agent with memory tools
agent = ReActAgent.from_tools(
    tools=[memory_tool, kg_tool],
    llm=Settings.llm,
    verbose=True,
)

Step 5: Test It

# Store a memory
vector_index.insert_nodes([TextNode(
    text="User mentioned they prefer Italian food and loved Mario's downtown",
    metadata={"user_id": "user_123", "context": "restaurant_recommendation"},
)])

# Later, recall it
response = agent.chat("What restaurant should I recommend to this user?")
print(response)

Best Practices

• Chunking matters: Split long conversations into smaller memory units. Store each fact or decision separately rather than entire conversation transcripts.
• Metadata is critical: Always store timestamps, user IDs, and context tags. Without metadata, memory retrieval becomes unreliable as the store grows.
• Use TTL for ephemeral memories: Not everything should live forever. Short-lived context (like “the user is currently looking at a flight page”) should expire.
• Index your memory index: When memory stores grow beyond 10K entries, build secondary indices (e.g., by topic or user) to speed up retrieval.
• Separate working memory from long-term memory: Working memory holds the current conversation context. Long-term memory holds facts learned over time. They serve different purposes.

Common Pitfalls

• Storing raw transcripts: Raw conversation logs are noisy. Extract facts and decisions, then store those.
• Forgetting to update the knowledge graph: When new information contradicts old facts, update the graph. Stale facts lead to wrong recommendations.
• Over-retrieving: Pulling 20 memories into context wastes tokens and confuses the model. Start with 3-5 highly relevant memories and expand only if needed.
• Ignoring privacy: Memory stores contain user data. Implement proper access controls, retention policies, and deletion capabilities.

Conclusion and Next Steps

Agent memory transforms stateless LLM interactions into persistent, context-aware assistants. Start with vector stores for semantic recall, add knowledge graphs when you need structured reasoning, and combine both for production systems.

The next frontier is memory consolidation — agents that periodically review and compress their memories, resolving contradictions and building coherent user models. LlamaIndex’s IngestionPipeline provides the building blocks for this.

Experiment with the code above, and you will quickly see how much more capable an agent becomes when it remembers.