Context Window Engineering for Long-Running Loops

Every model call in an agentic loop adds tokens to the conversation history. After enough steps, the accumulated context exceeds the model’s context window or degrades performance. This is the defining engineering constraint of long-running loops. This article covers techniques for keeping an agent’s state within budget without sacrificing performance.

The Context Window Challenge

Long-running loops degrade in a predictable pattern. Early steps are productive; the agent has fresh context, clear goals, and full access to relevant information. As steps accumulate, performance degrades. The agent loses track of early decisions, repeats work, misinterprets instructions, and produces increasingly incoherent output.

Context Drift

Context drift occurs when the agent’s focus shifts away from the original goal as new information accumulates. Each step adds observations, reasoning traces, and tool outputs. The signal-to-noise ratio declines: the goal, success criteria, and key constraints are diluted by the growing volume of intermediate state.

After 20 steps in a ReAct loop, the original user request occupies a tiny fraction of the context window. The model must sift through pages of tool outputs to remember what it was trying to accomplish.

Context Rot

Over many iterations, the context accumulates outdated or contradictory information. Early analysis may be superseded by later findings, but both remain in the context. Dead-end explorations, rejected approaches, and failed attempts persist alongside successful ones, confusing the model.

Degenerate Loops

A degenerate loop is one where context degradation causes performance to decline below a useful threshold while the loop continues running. The loop runs, but it is no longer converging.

def detect_degeneracy(step: int, recent_scores: list[float]) -> bool:
    if len(recent_scores) < 5:
        return False
    # declining trend over last 5 iterations
    return all(
        recent_scores[i] >= recent_scores[i + 1]
        for i in range(len(recent_scores) - 1)
    )

Context Pruning, Summarization, and Memory Compaction

Managing context in long-running loops requires a toolkit of strategies applied selectively.

Sliding Window

Keep only the last N messages. Cheap, predictable, and effective when only recent context matters, but suffers from catastrophic forgetting if early decisions are needed.

def sliding_window(messages: list[dict], window: int = 20) -> list[dict]:
    if len(messages) <= window:
        return messages
    system = [m for m in messages if m["role"] == "system"]
    recent = messages[-(window - len(system)):]
    return system + recent

Summarization

Compresses large blocks of context into compact summaries after every K steps:

def compress_context(
    messages: list[dict],
    compress_every: int = 10
) -> list[dict]:
    if len(messages) < compress_every:
        return messages

    to_compress = messages[-compress_every:-1]
    summary_prompt = f"Summarize the following interaction in 3 sentences: {to_compress}"
    summary = model.generate(summary_prompt)

    return messages[:-compress_every] + [
        {"role": "system", "content": f"[Summary] {summary}"}
    ] + [messages[-1]]

Hierarchical Memory

Separates context into three tiers:

• Working memory: Last N steps kept verbatim.
• Episodic memory: Summarized blocks of earlier steps.
• Semantic memory: Extracted persistent facts and decisions.

class HierarchicalMemory:
    def __init__(self, working_window: int = 20):
        self.working: list[dict] = []
        self.episodes: list[str] = []
        self.semantic: dict[str, str] = {}
        self.working_window = working_window

    def add(self, message: dict):
        self.working.append(message)
        if len(self.working) > self.working_window * 2:
            self._compress()

    def _compress(self):
        old = self.working[:self.working_window]
        summary = summarize(old)
        self.episodes.append(summary)
        self.working = self.working[self.working_window:]

    def get_context(self) -> list[dict]:
        context = []
        if self.episodes:
            context.append({"role": "system", "content": f"[Episodes]\n" + "\n".join(self.episodes)})
        if self.semantic:
            context.append({"role": "system", "content": f"[Facts]\n" + "\n".join(f"{k}: {v}" for k, v in self.semantic.items())})
        context.extend(self.working)
        return context

Worktrees & Environment Isolation

Execution environments accumulate state just like conversation context. Editing files, installing packages, and running commands cause environment rot.

Git Worktrees as Isolation Boundaries

Git worktrees provide isolated filesystem environments sharing the same repository storage:

# Create an isolated worktree for task execution
git worktree add ../worktree-fix-1 feature-branch

# Execute work in isolated directory
# Clean up when done
git worktree remove ../worktree-fix-1

Token Budgeting & Cost Optimization

Token budgeting treats tokens as a finite resource to be allocated across the loop’s lifecycle.

class CostBudget:
    def __init__(self, max_usd: float):
        self.max = max_usd
        self.spent = 0.0

    def record_call(self, model: str, input_tokens: int, output_tokens: int):
        cost = token_cost(model, input_tokens, output_tokens)
        self.spent += cost
        if self.spent > self.max:
            raise BudgetExceeded(f"${self.spent:.2f} > ${self.max:.2f}")

Cost-Aware Model Selection

Select models dynamically based on step complexity:

• Fast / Cheap Model: Routine actions, simple tool calls, parsing
• Capable / Advanced Model: Complex reasoning, architecture, planning
• Judge Model: Evaluation and validation

Key Takeaways

1. Detect context drift and rot early with automated degeneracy checks.
2. Implement hierarchical memory to balance detail with long-term retention.
3. Use Git worktrees to isolate execution environments and prevent filesystem contamination.
4. Enforce token and cost budgets with per-step limits and model switching.