Short-Term vs. Long-Term Memory: Architecting Multi-Horizon Recall

Effective agent memory isn’t one system — it’s multiple systems operating at different time horizons. Short-term memory handles the current conversation, medium-term memory spans recent sessions, and long-term memory persists indefinitely. Architecting these layers correctly determines whether an agent feels naturally aware or frustratingly forgetful.

The Memory Horizon Model

┌─────────────────────────────────────────────────┐
│  Working Memory (seconds to minutes)            │
│  Current prompt context, tool outputs           │
├─────────────────────────────────────────────────┤
│  Short-Term Memory (minutes to hours)           │
│  Current session, recent messages               │
├─────────────────────────────────────────────────┤
│  Medium-Term Memory (hours to days)             │
│  Recent sessions, active projects               │
├─────────────────────────────────────────────────┤
│  Long-Term Memory (days to forever)             │
│  Decisions, facts, permanent knowledge          │
└─────────────────────────────────────────────────┘

Working Memory

Horizon: Seconds to minutes Storage: In-context window (prompt) Purpose: Active task execution

Working memory is what’s currently in the LLM’s context window. It’s the fastest to access (zero latency) but limited by the model’s token limit.

class WorkingMemory:
    def __init__(self, max_tokens: int = 4000):
        self.max_tokens = max_tokens
        self.messages: list[dict] = []

    def add(self, message: dict):
        self.messages.append(message)
        self._enforce_budget()

    def _enforce_budget(self):
        while self._estimate_tokens() > self.max_tokens:
            # Remove oldest non-critical message
            self.messages.pop(0)

    def _estimate_tokens(self) -> int:
        return sum(len(m.get("content", "").split()) * 1.3
                   for m in self.messages)

    def get_context(self) -> list[dict]:
        return self.messages

Short-Term Memory

Horizon: Minutes to hours Storage: Fast key-value store (Redis) Purpose: Current session continuity

Short-term memory preserves context within a single session. When a user pauses and returns minutes later, short-term memory provides immediate context.

class ShortTermMemory:
    def __init__(self, session_id: str, ttl_seconds: int = 3600):
        self.session_id = session_id
        self.ttl = ttl_seconds
        self.store = RedisStore()

    def store_turn(self, turn: dict):
        key = f"session:{self.session_id}:turns"
        self.store.rpush(key, json.dumps(turn))
        self.store.expire(key, self.ttl)

    def get_recent_turns(self, n: int = 10) -> list[dict]:
        key = f"session:{self.session_id}:turns"
        raw = self.store.lrange(key, -n, -1)
        return [json.loads(r) for r in raw]

    def get_summary(self) -> str:
        key = f"session:{self.session_id}:summary"
        return self.store.get(key) or ""

Medium-Term Memory

Horizon: Hours to days Storage: Vector database with TTL Purpose: Active projects, recent context

Medium-term memory bridges sessions. It captures key information from recent interactions without overwhelming long-term storage.

class MediumTermMemory:
    def __init__(self, vector_store, ttl_days: int = 7):
        self.vector_store = vector_store
        self.ttl = timedelta(days=ttl_days)

    def store_session_summary(self, session_id: str, summary: str, metadata: dict):
        self.vector_store.upsert(
            id=f"medium:{session_id}",
            vector=self.embed(summary),
            payload={
                "content": summary,
                "type": "session_summary",
                "created_at": datetime.now().isoformat(),
                "expires_at": (datetime.now() + self.ttl).isoformat(),
                **metadata,
            }
        )

    def recall_relevant(self, query: str, top_k: int = 5) -> list[dict]:
        results = self.vector_store.search(
            vector=self.embed(query),
            top_k=top_k,
            filter={"type": {"$in": ["session_summary", "active_project"]}},
        )
        # Filter out expired entries
        now = datetime.now()
        return [
            r for r in results
            if datetime.fromisoformat(r["payload"]["expires_at"]) > now
        ]

Long-Term Memory

Horizon: Days to forever Storage: Vector DB + relational DB + graph DB Purpose: Permanent knowledge, decisions, facts

Long-term memory stores knowledge that should persist indefinitely. This includes user preferences, important decisions, extracted facts, and learned procedures.

class LongTermMemory:
    def __init__(self):
        self.vector_store = QdrantStore("long_term")
        self.kv_store = PostgresStore("long_term")
        self.graph_store = Neo4jStore()

    def store_fact(self, fact: KnowledgeFact):
        # Vector: for semantic search
        self.vector_store.upsert(
            id=f"fact:{fact.id}",
            vector=self.embed(fact.content),
            payload=fact.model_dump(),
        )

        # KV: for exact lookup
        self.kv_store.set(
            f"fact:{fact.id}",
            json.dumps(fact.model_dump()),
        )

        # Graph: for relationship tracking
        self.graph_store.create_fact_node(fact)

    def recall(self, query: str, time_horizon: str = "all") -> list[Memory]:
        if time_horizon == "all":
            # Search across all time horizons
            return self._unified_recall(query)
        elif time_horizon == "recent":
            # Only medium and short-term
            return self._search_with_filter(
                query,
                created_after=datetime.now() - timedelta(days=7)
            )
        elif time_horizon == "permanent":
            # Only long-term
            return self._search_with_filter(
                query,
                created_before=datetime.now() - timedelta(days=30)
            )

Unified Memory Interface

The key to multi-horizon memory is a unified interface that transparently searches across all layers:

class UnifiedMemory:
    def __init__(self):
        self.working = WorkingMemory(max_tokens=4000)
        self.short_term = ShortTermMemory(session_id="current")
        self.medium_term = MediumTermMemory(vector_store)
        self.long_term = LongTermMemory()

    def store(self, content: str, memory_type: str, metadata: dict):
        """Store in the appropriate memory layer."""
        memory = Memory(content=content, type=memory_type, metadata=metadata)

        # Always store in working memory
        self.working.add(memory.to_message())

        # Route to appropriate long-term store
        if memory_type == "session_turn":
            self.short_term.store_turn(memory.to_dict())
        elif memory_type == "session_summary":
            self.medium_term.store_session_summary(
                memory.content, metadata
            )
        elif memory_type in ("fact", "decision", "preference"):
            self.long_term.store_fact(memory)

    def recall(self, query: str, max_tokens: int = 4000) -> str:
        """Recall relevant memories across all layers."""
        candidates = []

        # Working memory (always included)
        candidates.extend(self.working.get_context())

        # Short-term memory
        candidates.extend(self.short_term.get_recent_turns(10))

        # Medium-term memory
        candidates.extend(self.medium_term.recall_relevant(query, 5))

        # Long-term memory
        candidates.extend(self.long_term.recall(query, 10))

        # Rank and select within token budget
        ranked = self._rank_by_relevance(candidates, query)
        selected = self._select_within_budget(ranked, max_tokens)

        return self._format_context(selected)

    def _rank_by_relevance(self, candidates, query):
        """Score and rank candidates by relevance to query."""
        scored = []
        for c in candidates:
            score = self._calculate_score(c, query)
            scored.append((c, score))
        return [c for c, s in sorted(scored, key=lambda x: x[1], reverse=True)]

    def _select_within_budget(self, ranked, max_tokens):
        """Select memories within token budget."""
        selected = []
        tokens_used = 0
        for memory in ranked:
            tokens = estimate_tokens(memory)
            if tokens_used + tokens <= max_tokens:
                selected.append(memory)
                tokens_used += tokens
        return selected

Recall Strategy by Task Type

Task Type	Primary Horizon	Secondary Horizon
Answering current question	Working + Short-term	Medium-term
Continuing recent project	Short-term + Medium	Long-term
Referencing past decision	Long-term	Medium-term
Debugging an issue	Short-term	Long-term
Building on past work	All horizons	—

Best Practices

1. Separate by time, not by type — Different memory types (facts, decisions, conversations) can coexist in the same layer. The key separation is time horizon.

2. Promote memories across layers — When a short-term memory proves valuable (referenced multiple times, marked as important), promote it to medium or long-term storage.

3. Match retrieval to horizon — Don’t search long-term memory for current session context. Use the appropriate layer for each query type.

4. Monitor layer utilization — Track how often each memory layer is accessed. If long-term memory is rarely used, you may be over-investing in storage.

5. Test across horizons — Verify that your memory system works correctly across all time horizons. A memory system that only works for current sessions is incomplete.

Conclusion

Multi-horizon memory architecture gives agents the right information at the right time. Working memory handles active tasks, short-term memory maintains session continuity, medium-term memory bridges recent sessions, and long-term memory preserves permanent knowledge. By separating these layers and providing a unified recall interface, you create agents that are both responsive to immediate context and aware of their full history.

References:

• OpenAI Agents SDK Sessions
• LangGraph Stateful Agent Memory