Long-Term Knowledge Stores: Moving Beyond Rolling Chat History

The default memory strategy for most LLM applications is a rolling chat history — keep the last N messages and discard the rest. This works for simple chatbots, but it fails catastrophically for agents that need to accumulate knowledge over time. When a user asks “What did we decide about the database migration last month?”, a rolling window agent has no answer. Long-term knowledge stores solve this by persisting structured information across the entire lifecycle of an agent.

The Problem with Rolling Chat History

Rolling chat history has three fundamental limitations:

1. Fixed context window — Once messages fall outside the window, they’re gone forever. An agent that has helped you with 100 projects only remembers the last 2-3.

2. No structure — Raw message logs mix important decisions with casual pleasantries. There’s no way to query “what were the key decisions?” without re-reading everything.

3. No synthesis — Rolling history stores verbatim text, not extracted knowledge. The insight “the client prefers PostgreSQL over MySQL” is buried in a 200-message conversation about database options.

Long-Term Knowledge Store Architecture

A proper long-term knowledge store separates knowledge into distinct layers:

┌─────────────────────────────────────────┐
│           Working Memory                 │
│  (Current session context)              │
├─────────────────────────────────────────┤
│           Episodic Memory                │
│  (Past sessions and interactions)       │
├─────────────────────────────────────────┤
│           Semantic Memory                │
│  (Facts, preferences, decisions)        │
├─────────────────────────────────────────┤
│           Procedural Memory             │
│  (How to perform tasks)                 │
└─────────────────────────────────────────┘

Semantic Memory: Extracted Facts

Semantic memory stores structured facts extracted from conversations:

from pydantic import BaseModel
from typing import Optional
from datetime import datetime

class KnowledgeFact(BaseModel):
    fact_id: str
    content: str  # "Client prefers PostgreSQL"
    category: str  # "preference", "decision", "constraint"
    confidence: float  # 0.0 to 1.0
    source_session: str
    created_at: datetime
    last_validated: Optional[datetime] = None
    tags: list[str] = []

Building with LlamaIndex

LlamaIndex provides tools for building structured knowledge stores:

from llama_index.core import VectorStoreIndex, Document
from llama_index.core.extractors import (
    TitleExtractor,
    QuestionsAnsweredExtractor,
)
from llama_index.core.ingestion import IngestionPipeline

class LongTermKnowledgeStore:
    def __init__(self, vector_store):
        self.index = VectorStoreIndex.from_vector_store(vector_store)
        self.facts: dict[str, KnowledgeFact] = {}

    def ingest_session(self, messages: list[dict], session_id: str):
        """Process a session and extract knowledge."""
        # Convert messages to documents
        documents = [
            Document(
                text=msg["content"],
                metadata={
                    "session_id": session_id,
                    "role": msg["role"],
                    "timestamp": msg.get("timestamp"),
                }
            )
            for msg in messages
        ]

        # Extract knowledge
        pipeline = IngestionPipeline(
            transformations=[
                QuestionsAnsweredExtractor(questions=3),
                TitleExtractor(),
            ]
        )

        processed_docs = pipeline.run(documents=documents)
        self.index.insert_nodes(processed_docs)

    def query_knowledge(self, query: str, top_k: int = 5) -> list[dict]:
        """Query the knowledge store."""
        query_engine = self.index.as_query_engine(
            similarity_top_k=top_k
        )
        response = query_engine.query(query)
        return [
            {
                "content": node.text,
                "score": node.score,
                "metadata": node.metadata,
            }
            for node in response.source_nodes
        ]

    def extract_facts(self, session_id: str) -> list[KnowledgeFact]:
        """Extract structured facts from a session."""
        # Query for decisions, preferences, and constraints
        fact_queries = [
            "What decisions were made?",
            "What preferences were expressed?",
            "What constraints or requirements were identified?",
        ]

        facts = []
        for query in fact_queries:
            results = self.query_knowledge(
                query,
                top_k=3
            )
            for result in results:
                fact = KnowledgeFact(
                    fact_id=str(uuid4()),
                    content=result["content"],
                    category=self._categorize_fact(query),
                    confidence=result["score"],
                    source_session=session_id,
                    created_at=datetime.now(),
                )
                facts.append(fact)
                self.facts[fact.fact_id] = fact

        return facts

Querying Across Sessions

The real power of long-term knowledge stores is cross-session querying:

# User asks about a decision from months ago
results = store.query_knowledge(
    "What database did we choose for the payment service?"
)

# Agent can now reference past context
response = f"""
Based on our previous discussions, you decided to use PostgreSQL
for the payment service. Key factors in that decision:
- {results[0]['content']}
- {results[1]['content']}
"""

Implementation Patterns

Pattern 1: Automatic Fact Extraction

After every session, automatically extract and store key facts:

async def on_session_end(session_messages, store):
    facts = store.extract_facts(session_id)
    for fact in facts:
        if fact.confidence > 0.7:
            await notify_user_about_new_fact(fact)

Pattern 2: Knowledge Validation

Periodically validate that stored facts are still true:

async def validate_facts(store, llm):
    for fact_id, fact in store.facts.items():
        is_still_valid = await llm.check_fact_relevance(
            fact.content,
            current_context
        )
        if not is_still_valid:
            fact.confidence *= 0.5  # Decay confidence
            if fact.confidence < 0.1:
                store.archive_fact(fact_id)

Pattern 3: Contextual Retrieval

Combine long-term knowledge with current session context:

def build_enriched_context(store, current_session):
    # Get recent context from current session
    recent = current_session.get_recent_messages(5)

    # Get relevant long-term knowledge
    query = " ".join([m["content"] for m in recent[-3:]])
    long_term = store.query_knowledge(query, top_k=5)

    return {
        "recent_context": recent,
        "long_term_knowledge": long_term,
    }

Storage Backend Comparison

Backend	Best For	Trade-offs
Vector DB	Semantic search	Slower exact queries
Key-Value	Exact lookups	No semantic search
Graph DB	Relationship traversal	Higher complexity
Relational DB	Structured queries	Less flexible schema

For most agent applications, a hybrid approach works best: use a vector store for semantic retrieval, backed by a relational database for structured queries and metadata filtering.

Best Practices

1. Extract facts, not raw text — Don’t just store conversations. Use the LLM to extract structured knowledge (decisions, preferences, constraints) from each session.

2. Track provenance — Always record where a fact came from (which session, which user, when). This enables validation and debugging.

3. Decay confidence over time — Facts can become outdated. Implement confidence decay so old, unvalidated facts don’t dominate retrieval results.

4. Separate knowledge types — Keep facts, episodes, and procedures in different stores or collections. They have different retrieval patterns and update frequencies.

5. Enable user control — Let users view, edit, and delete stored knowledge. Transparency builds trust.

Conclusion

Long-term knowledge stores are essential for agents that need to maintain context across sessions. By moving beyond rolling chat history and implementing structured knowledge extraction, storage, and retrieval, you can build agents that truly learn and accumulate expertise over time. The investment in proper memory architecture pays dividends in user experience and agent capability.

References:

• LlamaIndex Storing Modules
• LlamaIndex Agent Memory