Episodic Memory in Agents: Recording Sessions as Replayable Experiences

Most agents forget everything the moment a session ends. The next time a similar task arrives, the agent starts from scratch — re-discovering the same patterns, making the same mistakes, and requiring the same human corrections. Episodic memory solves this by recording entire agent sessions as structured, replayable experiences that the agent can reference and learn from over time.

What Is Episodic Memory?

Episodic memory is a memory system inspired by cognitive science that stores specific episodes — complete sequences of events, decisions, and outcomes. Unlike semantic memory (which stores facts) or procedural memory (which stores skills), episodic memory captures the full narrative of what happened during a particular interaction.

Episode = {
    context: { task, user, environment },
    steps: [ action1, observation1, action2, ... ],
    outcome: { success, metrics, artifacts },
    reflection: { what_worked, what_failed, lessons }
}

In agent systems, this means recording not just the final answer, but the entire trajectory: what the agent tried, what tools it called, what errors it encountered, and how it recovered.

Why Agents Need Episodic Memory

Learning from Past Mistakes

Without episodic memory, an agent that fails at a task has no way to avoid the same failure next time. With episodic memory, the agent can recall: “Last time I tried to deploy this service, the build failed because of a missing environment variable. I should check for that first.”

Accelerating Familiar Tasks

When an agent encounters a task it has solved before, episodic memory lets it skip the exploration phase and jump straight to a known-good solution path. This can reduce task completion time by 50-80% for recurring workflows.

Enabling Continuous Improvement

Episodic memories serve as a dataset for agent improvement. By analyzing patterns across many episodes, developers can identify systematic issues, optimize tool selection, and refine agent instructions.

Implementation with LlamaIndex

LlamaIndex provides first-class support for agent memory through its memory modules. Here’s how to build an episodic memory system:

Defining the Episode Schema

from pydantic import BaseModel, Field
from typing import List, Optional
from datetime import datetime

class AgentStep(BaseModel):
    action: str
    tool_used: Optional[str] = None
    input_data: dict = {}
    output_data: dict = {}
    timestamp: datetime = Field(default_factory=datetime.now)

class Episode(BaseModel):
    episode_id: str
    task_description: str
    steps: List[AgentStep] = []
    outcome: str  # "success" | "failure" | "partial"
    lessons_learned: List[str] = []
    context_tags: List[str] = []
    created_at: datetime = Field(default_factory=datetime.now)

Building the Episodic Memory Store

from llama_index.core import VectorStoreIndex
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.vector_stores import SimpleVectorStore

class EpisodicMemory:
    def __init__(self):
        self.episodes: List[Episode] = []
        self.index: Optional[VectorStoreIndex] = None

    def record_episode(self, episode: Episode):
        self.episodes.append(episode)
        self._update_index()

    def recall_similar(self, task: str, top_k: int = 3) -> List[Episode]:
        if not self.index:
            return []
        query_engine = self.index.as_query_engine()
        response = query_engine.query(
            f"Find episodes similar to: {task}"
        )
        return response.source_nodes

    def _update_index(self):
        documents = [
            self._episode_to_doc(ep) for ep in self.episodes
        ]
        self.index = VectorStoreIndex.from_documents(documents)

    def _episode_to_doc(self, episode: Episode):
        from llama_index.core import Document
        content = f"""
        Task: {episode.task_description}
        Steps: {len(episode.steps)}
        Outcome: {episode.outcome}
        Lessons: {'; '.join(episode.lessons_learned)}
        Tags: {', '.join(episode.context_tags)}
        """
        return Document(text=content, metadata=episode.model_dump())

Using Episodic Memory in an Agent Loop

class EpisodicAgent:
    def __init__(self, memory: EpisodicMemory, llm):
        self.memory = memory
        self.llm = llm

    async def run_task(self, task: str) -> str:
        # Recall relevant past episodes
        similar_episodes = self.memory.recall_similar(task)

        # Inject episode context into prompt
        episode_context = self._format_episodes(similar_episodes)
        prompt = f"""
        Task: {task}

        Relevant past experiences:
        {episode_context}

        Use these past experiences to guide your approach.
        Record your steps so we can learn from this episode.
        """

        # Execute the task
        result = await self.llm.complete(prompt)

        # Record this episode
        episode = Episode(
            episode_id=str(uuid4()),
            task_description=task,
            steps=self._extract_steps(result),
            outcome="success",
            lessons_learned=self._extract_lessons(result)
        )
        self.memory.record_episode(episode)

        return result

    def _format_episodes(self, episodes):
        return "\n".join([
            f"- Task: {ep.task_description} | Outcome: {ep.outcome} | "
            f"Lessons: {', '.join(ep.lessons_learned)}"
            for ep in episodes
        ])

Storage Backends for Episodic Memory

Vector Store (Semantic Search)

Best for finding episodes by semantic similarity. Use when you need “find me episodes related to this type of task.”

from llama_index.vector_stores.qdrant import QdrantVectorStore

vector_store = QdrantVectorStore(
    collection_name="episodes",
    url="http://localhost:6333"
)

Key-Value Store (Exact Lookup)

Best for retrieving specific episodes by ID or metadata. Use when you need “get me episode #1234.”

from llama_index.core.storage.kvstore import SimpleKVStore

kv_store = SimpleKVStore()
kv_store.put("episode:1234", episode.model_dump())

Graph Store (Relationship Tracing)

Best for exploring causal chains between episodes. Use when you need “show me all episodes that led to this outcome.”

Best Practices

1. Record liberally — Capture every step, even failed ones. Failed episodes are often the most valuable for learning.

2. Extract lessons explicitly — Don’t just store raw steps. Use the LLM to generate structured reflections on what worked and what didn’t.

3. Tag episodes with context — Include metadata like task type, tools used, user, and environment. This makes retrieval more precise.

4. Prune old episodes — Not all episodes remain relevant forever. Implement TTL-based or relevance-based pruning to keep the memory store manageable.

5. Index for multiple retrieval modes — Combine vector search (for semantic similarity) with metadata filtering (for exact matches) to cover different recall needs.

Conclusion

Episodic memory transforms agents from stateless executors into learning systems that accumulate experience over time. By recording complete sessions as structured episodes, agents can recall past successes, avoid repeated failures, and continuously improve their performance. With frameworks like LlamaIndex providing the building blocks, implementing episodic memory is now a practical reality for production agent systems.

References:

• LlamaIndex Agent Memory Documentation
• LlamaIndex Storing Modules