Skip to content
Blog

Context Compaction and Eviction: Managing Long Agent Histories

Long-running agents blow past their context windows. Learn eviction policies, recursive summarization checkpoints, and token-budget math, with LangChain trimming utilities.

Published on • November 16, 2026

AI Assistant

Your agent has been working for forty minutes on a migration plan. It has read 30 files and run 14 commands, and its history just crossed 180,000 tokens. Then the provider returns 400 invalid_request_error: context length exceeded — and because nothing in your stack expected it, the run dies with all of that reasoning gone.

Even when it doesn’t error, long context quietly degrades you: responses slow down, costs climb linearly, and models “get distracted” by stale or off-topic content far earlier than the advertised window suggests. The LangChain team puts it bluntly in their short-term memory docs: most LLMs “still perform poorly over long contexts.”

In this post, you will learn how to keep an agent running indefinitely: token accounting with budget alarms, eviction policies (FIFO, importance/recency scoring, tool-result-first trimming), recursive summarization with compaction checkpoints, and structured summaries as durable memory — implemented in runnable Python, then mapped onto LangChain’s trim_messages and SummarizationMiddleware so you can adopt the battle-tested versions. The current reference for those utilities lives in the LangChain short-term memory guide.

Prerequisites

  • Python 3.10+, pip install langchain langgraph tiktoken.
  • An agent you already have that keeps a message list (any framework; the algorithm is framework-agnostic).
  • Familiarity with chat message roles: system, human/user, AI/assistant, tool.

Count Before You Compress: Token Accounting

Compaction without measurement is guesswork. You need three numbers per turn:

from dataclasses import dataclass

import tiktoken


def approx_tokens(text: str) -> int:
    """Cheap token estimate; swap in your provider's tokenizer for accuracy."""
    return len(tiktoken.get_encoding("o200k_base").encode(text))


def message_tokens(message: dict) -> int:
    """Tokens for one message: role marker + content + tool payload."""
    content = message.get("content") or ""
    if isinstance(content, list):  # multimodal blocks
        content = " ".join(str(c.get("text", "")) for c in content)
    total = approx_tokens(str(content)) + 4  # role/formatting overhead
    for call in message.get("tool_calls") or []:
        total += approx_tokens(str(call))
    return total


def history_tokens(messages: list[dict]) -> int:
    return sum(message_tokens(m) for m in messages)


@dataclass
class Budget:
    window: int = 128_000      # model's total context
    reserve_output: int = 8_000  # max_tokens for the reply
    soft_ratio: float = 0.70    # compact when history crosses this share
    hard_ratio: float = 0.85    # alarm: evict immediately or the call fails

    @property
    def usable(self) -> int:
        return self.window - self.reserve_output

    @property
    def soft_limit(self) -> int:
        return int(self.usable * self.soft_ratio)

    @property
    def hard_limit(self) -> int:
        return int(self.usable * self.hard_ratio)

The two limits do different jobs. The soft limit is where compaction triggers calmly before a call, leaving room to think. The hard limit is a budget alarm: log it, page someone, evict synchronously — the next call may fail outright. Emitting agent.context.utilization at 70% and 85% turns a silent failure into a dashboard signal.

Eviction Policies: What Goes First

Eviction means dropping messages outright: the cheapest intervention, and the one most likely to remove something important. The policy matters:

  • FIFO. Drop the oldest first. Trivial, and fine for chat where early smalltalk is worthless — but dangerous for agents, because the original task statement lives at position zero.
  • Importance/recency scoring. Score each message and drop the lowest: score = 0.6 * recency + 0.4 * importance. Importance comes from signals you already have — the original request, approvals, errors that changed the plan. Pin anything above a threshold.
  • Tool-result-first trimming. Tool messages are the largest objects in the history and the first to go stale — the file you read at turn 3 is not news at turn 40. Replace a 6,000-token result with a 40-token stub keeping the id and a digest; leave the original in a side store.
import math


def score_message(msg: dict, index: int, total: int,
                  pinned_ids: set[str]) -> float:
    if msg.get("id") in pinned_ids:
        return math.inf  # never evicted: original request, approvals, final plan
    recency = index / max(1, total - 1)          # 0 (oldest) .. 1 (newest)
    importance = 0.0
    if msg.get("role") == "user":
        importance += 0.5
    if msg.get("role") == "tool" and msg.get("is_error"):
        importance += 0.3  # failures explain why the plan changed
    if msg.get("role") == "system":
        importance += 1.0
    return 0.6 * recency + 0.4 * importance


def evict_by_score(messages: list[dict], budget: int,
                   pinned_ids: set[str]) -> list[dict]:
    """Remove lowest-scoring messages until history fits `budget`.
    Within a score band, tool messages go first — they are recoverable
    by re-running the tool.
    """
    if history_tokens(messages) <= budget:
        return messages

    ranked = sorted(
        range(len(messages)),
        key=lambda i: (score_message(messages[i], i, len(messages), pinned_ids),
                       0 if messages[i].get("role") == "tool" else 1),
    )
    drop, kept = set(), history_tokens(messages)
    for i in ranked:
        if kept <= budget:
            break
        drop.add(i)
        kept -= message_tokens(messages[i])
    return [m for i, m in enumerate(messages) if i not in drop]

Eviction is lossy: it buys headroom, it does not preserve knowledge. That is what compaction is for.

Compaction: Recursive Summarization With Checkpoints

Compaction replaces a block of history with a summary of it. The recursive part is what makes it survive a run longer than one compaction: each cycle summarizes the previous summary plus the newest eligible messages, so memory grows instead of being rewritten from scratch. summarize_fn below is simply your LLM call.

def compact_once(messages: list[dict], summary: str,
                 keep_recent: int, summarize_fn) -> tuple[list[dict], str]:
    """One compaction step: fold everything but the newest `keep_recent`
    messages into `summary` and return the surviving history."""
    if len(messages) <= keep_recent + 1:
        return messages, summary

    head, tail = messages[:-keep_recent], messages[-keep_recent:]

    prompt = (
        "You are compacting an agent's working memory. Produce a terse, "
        "durable summary preserving: the original goal, decisions and why, "
        "key entities/ids, tool results that changed the plan, errors and "
        "their fixes, and open TODOs. Do not invent facts.\n\n"
        f"PREVIOUS SUMMARY:\n{summary or '(none)'}\n\n"
        "MESSAGES TO FOLD IN:\n"
        + "\n".join(
            f"[{m.get('role')}] {str(m.get('content'))[:1500]}" for m in head
        )
    )
    new_summary = summarize_fn(prompt)

    # Checkpoint before mutating: recovery point if the next model call
    # fails or the summary turns out to be garbage.
    persist_checkpoint({"summary": new_summary, "folded": len(head),
                        "tokens_before": history_tokens(messages)})

    compacted = [
        {"role": "system", "content": f"[COMPACTED CONTEXT]\n{new_summary}"},
        {"role": "user", "content": "[The earlier conversation was compacted. "
                                    "Treat the context above as authoritative history.]"},
        *tail,
    ]
    return compacted, new_summary


def persist_checkpoint(checkpoint: dict) -> None:
    """Durable and re-readable: Redis, Postgres, or a file."""
    import json, pathlib
    pathlib.Path("checkpoints").mkdir(exist_ok=True)
    (pathlib.Path("checkpoints") / f"ckpt_{checkpoint['folded']}.json").write_text(
        json.dumps(checkpoint, indent=2)
    )

Three design choices to steal: the summary lives in a system-role message, so it reads as instruction-grade context; the newest turns stay verbatim, preserving the working set; and a checkpoint is written before history is mutated, so a bad summary or a crashed call never loses the run.

The Compaction Loop With Budget Math

The loop that runs before every model call:

def prepare_context(messages: list[dict], summary: str, budget: Budget,
                     summarize_fn) -> tuple[list[dict], str, dict]:
    """Before each model invoke. Returns (messages, summary, stats)."""
    used = history_tokens(messages)
    stats = {"tokens_before": used, "alarm": False, "action": "none"}

    if used > budget.hard_limit:
        stats["alarm"] = True           # budget alarm: metric + log here
        stats["action"] = "hard_evict"
        messages = evict_by_score(messages, budget.soft_limit,
                                  pinned_ids=pinned(messages))
        used = history_tokens(messages)

    if used > budget.soft_limit:
        stats["action"] = "compact"
        keep_recent = 8                  # newest turns stay verbatim
        messages, summary = compact_once(
            messages, summary, keep_recent, summarize_fn
        )
        if history_tokens(messages) > budget.soft_limit:  # huge tail
            messages = evict_by_score(messages, budget.soft_limit,
                                      pinned_ids=pinned(messages))

    stats["tokens_after"] = history_tokens(messages)
    stats["utilization"] = round(stats["tokens_after"] / budget.usable, 3)
    return messages, summary, stats


def pinned(messages: list[dict]) -> set[str]:
    """Anchors: the system prompt plus the first user turn (the task)."""
    ids = {m["id"] for m in messages if m.get("role") == "system" and m.get("id")}
    for m in messages:
        if m.get("role") == "user":
            ids.add(m.get("id"))
            break
    return ids

Worked example: 128k window, 8k reserved for output → usable = 120,000, soft limit 84,000, hard limit 102,000. A 110,000-token history trips the alarm, is evicted to ~84k, then folds the first 60 messages into a ~1,500-token summary with the newest 8 turns verbatim — landing near 26,000 tokens (22% utilization).

LangChain’s Utilities: Don’t Roll Your Own in Production

Trimming — langchain_core.messages.trim_messages takes a strategy ("last", "first", "middle") plus max_tokens/max_messages. In an agent, wire it into @before_model middleware so it runs before every model call:

from langchain.agents import create_agent, AgentState
from langchain.agents.middleware import before_model
from langchain.messages import RemoveMessage
from langgraph.graph.message import REMOVE_ALL_MESSAGES


@before_model
def keep_recent(state: AgentState) -> dict | None:
    """Keep the first message plus the last few — classic anchor + recency."""
    messages = state["messages"]
    if len(messages) <= 6:
        return None
    new = [messages[0], *messages[-5:]]
    return {"messages": [RemoveMessage(id=REMOVE_ALL_MESSAGES), *new]}

Summarization — for the compaction path, SummarizationMiddleware does the recursive job out of the box:

from langchain.agents.middleware import SummarizationMiddleware

agent = create_agent(
    model="your-model",
    tools=[...],
    middleware=[
        SummarizationMiddleware(
            model="a-cheap-summarizer-model",
            trigger=("tokens", 40000),   # soft limit
            keep=("messages", 20),       # verbatim tail
        )
    ],
    checkpointer=checkpointer,
)

The validity rule you must not break. After trimming or summarizing, the history must stay valid for your provider: some require it to start with a user message, most require an assistant message with tool calls to be followed by its matching tool messages. An orphaned tool_calls block fails the very API call you were protecting. Always run validate_history() after compaction:

def validate_history(messages: list[dict]) -> list[dict]:
    """Drop orphaned tool_calls so the request stays provider-legal."""
    for i, m in enumerate(messages):
        if m.get("tool_calls"):
            nxt = messages[i + 1] if i + 1 < len(messages) else {}
            if nxt.get("role") != "tool":
                m["tool_calls"] = []
    return messages

Structured Summaries as Durable Memory

A paragraph summary is good for continuity; a structured one is good for everything else — queryable, diffable, promotable into long-term memory when the session ends:

{
  "goal": "Migrate billing service to Postgres 16",
  "decisions": [{"choice": "dual-write for 48h", "why": "zero-downtime requirement"}],
  "entities": {"service": "billing-api", "db": "pg16-prod"},
  "tool_results": [{"tool": "run_sql", "digest": "3.2M rows in billing_invoices", "reusable": true}],
  "errors": [{"what": "pool exhaustion", "fix": "max_conns 20 -> 60"}],
  "todos": ["backfill ledger table", "verify replication lag < 1s"]
}

Checkpoint it, and at session end write it to your long-term store: injecting goal/decisions/todos next session is eviction-proof, because state outside the window cannot be evicted from it.

Putting It All Together

A 90-minute run now looks like this:

turn  40   87,000 tok  (73%)  → above soft limit: compact; summary #1 =
                                1,400 tok, tail of 8 kept → 24,000 tok (20%)
turn  95   86,000 tok  (72%)  → compact again: summary #1 + 60 msgs →
                                summary #2 → 31,000 tok (26%)
turn 120  104,000 tok  (87%)  → HARD ALARM: evict lowest-score messages
                                to 84,000, then compact → 29,000 tok
turn 180   final answer from summary #2 + recent turns + pinned task

The run never touches the context limit, the original task survives to turn 180 because it was pinned, and the checkpoint trail lets you replay what the model saw at any point.

Conclusion and Next Steps

You now have a complete long-horizon memory strategy: measure tokens and alarm at 70%/85%, evict the cheapest thing first (scoring, with tool results taking the hit), compact recursively into a checkpointed summary keeping the newest turns verbatim, and validate that the compressed history is provider-legal. LangChain’s trim_messages, @before_model, and SummarizationMiddleware are the production versions of this loop.

Homework:

  1. Log tokens_before/tokens_after/action for every turn in a real run — the distribution tells you whether keep_recent=8 fits your workload.
  2. Add a reusable: true flag to tool-result digests and re-run those tools on demand instead of storing their output at all.
  3. A/B test trimming against summarization on a 20-task eval set, then promote goal/decisions/todos into long-term memory so a new session can resume a 90-minute run in one turn.