Conversation Summaries as Memory: Compressing Sessions for Context

Long conversations eat context windows. A 50-message interaction can consume 20,000+ tokens, leaving little room for the actual task. Conversation summaries solve this by compressing lengthy sessions into concise representations that preserve the essential information while dramatically reducing token count. For agents running long workflows, this isn’t optional — it’s essential.

Why Summarize Conversations?

Token Budget Management

Every LLM has a finite context window. Without summarization, agents working on multi-step tasks quickly run out of space:

Without Summaries:
  System prompt:     2,000 tokens
  Conversation:     25,000 tokens (50 messages)
  Available for task: 3,000 tokens (out of 30K window)
  
With Summaries:
  System prompt:     2,000 tokens
  Summary:           1,500 tokens (compressed from 50 messages)
  Available for task: 26,500 tokens

Context Continuity

Summaries maintain context across session boundaries. When a user returns hours or days later, the summary provides immediate context without requiring them to repeat everything.

Cost Reduction

Fewer input tokens means lower API costs. Summarizing a 50-message conversation into a 200-word summary can reduce costs by 80-90%.

Implementation Strategies

Strategy 1: Sliding Window Summarization

Summarize older messages as new ones arrive:

from typing import List, Dict
import asyncio

class SlidingWindowSummarizer:
    def __init__(self, llm, max_window_size: int = 10):
        self.llm = llm
        self.max_window_size = max_window_size
        self.messages: List[Dict] = []
        self.summary: str = ""

    async def add_message(self, message: Dict):
        self.messages.append(message)

        if len(self.messages) > self.max_window_size:
            await self._compress_old_messages()

    async def _compress_old_messages(self):
        # Split into old messages to summarize and recent to keep
        old_messages = self.messages[:-5]
        recent_messages = self.messages[-5:]

        # Generate summary of old messages
        new_summary = await self._summarize(
            self.summary,
            old_messages
        )

        self.summary = new_summary
        self.messages = recent_messages

    async def _summarize(self, existing_summary: str, messages: List[Dict]) -> str:
        conversation = "\n".join([
            f"{m['role']}: {m['content']}" for m in messages
        ])

        prompt = f"""
        Summarize this conversation, preserving key decisions, 
        facts, and context. Be concise but complete.

        Previous summary: {existing_summary}

        New messages:
        {conversation}

        Updated summary:
        """

        response = await self.llm.complete(prompt)
        return response.text

    def get_context(self) -> str:
        if self.summary:
            return f"Previous context: {self.summary}\n\nRecent messages:\n" + \
                   "\n".join([f"{m['role']}: {m['content']}" for m in self.messages])
        return "\n".join([f"{m['role']}: {m['content']}" for m in self.messages])

Strategy 2: Hierarchical Summaries

Create summaries at multiple levels of detail:

class HierarchicalSummarizer:
    def __init__(self, llm):
        self.llm = llm
        self.level_0_summary = ""  # One-liner
        self.level_1_summary = ""  # Paragraph
        self.level_2_summary = ""  # Detailed sections

    async def update(self, messages: List[Dict]):
        # Update all levels
        self.level_0_summary = await self._summarize_level(
            messages, "one sentence"
        )
        self.level_1_summary = await self._summarize_level(
            messages, "one paragraph"
        )
        self.level_2_summary = await self._summarize_level(
            messages, "detailed sections"
        )

    async def _summarize_level(self, messages, detail_level):
        conversation = "\n".join([
            f"{m['role']}: {m['content']}" for m in messages
        ])

        prompt = f"""
        Create a {detail_level} summary of this conversation.
        Preserve: decisions made, key facts, action items, 
        and current state.

        {conversation}
        """

        response = await self.llm.complete(prompt)
        return response.text

    def get_context(self, depth: str = "medium") -> str:
        if depth == "brief":
            return self.level_0_summary
        elif depth == "medium":
            return self.level_1_summary
        else:
            return self.level_2_summary

Strategy 3: Extractive Summarization

Pull key sentences rather than generating new text:

class ExtractiveSummarizer:
    def __init__(self, llm):
        self.llm = llm

    async def extract_key_points(self, messages: List[Dict]) -> List[str]:
        conversation = "\n".join([
            f"{m['role']}: {m['content']}" for m in messages
        ])

        prompt = f"""
        Extract the key points from this conversation. Return 
        each as a concise sentence.

        {conversation}

        Key points:
        """

        response = await self.llm.complete(prompt)
        return [line.strip() for line in response.text.split("\n") if line.strip()]

Summarization Prompts That Work

For Technical Discussions

Summarize this technical conversation, preserving:
- All decisions made and their rationale
- Technical specifications and constraints
- Action items and owners
- Open questions or blockers
- Code snippets or commands discussed

For Customer Support

Summarize this support interaction, preserving:
- Customer's issue and symptoms
- Troubleshooting steps attempted
- Resolution (if reached)
- Customer sentiment
- Follow-up items

For Research Sessions

Summarize this research discussion, preserving:
- Questions being investigated
- Sources consulted
- Findings and evidence
- Conclusions drawn
- Remaining research questions

Advanced: Topic-Aware Summarization

Segment conversations by topic and summarize each separately:

class TopicAwareSummarizer:
    def __init__(self, llm):
        self.llm = llm

    async def segment_by_topic(self, messages: List[Dict]) -> Dict[str, List[Dict]]:
        """Group messages by topic."""
        prompt = f"""
        Analyze these messages and group them by topic.
        Return a JSON mapping of topic names to message indices.

        Messages:
        {self._format_messages(messages)}
        """

        response = await self.llm.complete(prompt)
        return json.loads(response.text)

    async def summarize_by_topic(self, messages: List[Dict]) -> Dict[str, str]:
        """Create separate summaries for each topic."""
        segments = await self.segment_by_topic(messages)
        summaries = {}

        for topic, topic_messages in segments.items():
            summaries[topic] = await self._summarize_topic(
                topic, topic_messages
            )

        return summaries

    def build_context(self, summaries: Dict[str, str]) -> str:
        return "\n\n".join([
            f"## {topic}\n{summary}"
            for topic, summary in summaries.items()
        ])

Best Practices

1. Summarize incrementally — Don’t wait until the context window is full. Summarize periodically (every 10-20 messages) to keep summaries fresh and manageable.

2. Preserve structure — Use bullet points, headers, and sections in summaries. Structured summaries are easier to scan and extract information from.

3. Include metadata — Add timestamps, speaker names, and message counts to summaries for better traceability.

4. Test summary quality — Periodically verify that summaries contain the key information from the original conversations. Use LLM-as-a-judge or human review.

5. Version summaries — Keep historical summaries so you can trace how understanding evolved over time.

Conclusion

Conversation summaries are essential for agents that need to maintain context across long interactions or sessions. By compressing verbose conversations into concise, structured summaries, agents preserve critical information while staying within token budgets. The right summarization strategy depends on your use case — sliding window for ongoing conversations, hierarchical for multi-depth needs, and topic-aware for complex multi-subject discussions.

References:

• Gemini API Context Management
• OpenAI Agents SDK Sessions