Memory-Augmented Prompts: Injecting Recall into Every Agent Turn

Static prompts are stateless. Every turn starts fresh, with no awareness of what happened before. Memory-augmented prompts change this by automatically injecting relevant past context into every interaction. The agent doesn’t need to be told to remember — it remembers because the prompt system handles recall transparently.

The Problem with Static Prompts

Without memory injection, agents lose context between turns:

Turn 1: "My name is Alice and I prefer PostgreSQL."
Turn 2: "What database should I use?"
Agent: "It depends on your requirements..."  (Forgets Alice's preference)

Memory-augmented prompts solve this by dynamically building the prompt with relevant context:

System: You are a helpful assistant.
Memory Context: User's name is Alice. User prefers PostgreSQL.
User: "What database should I use?"
Agent: "Based on your preference for PostgreSQL, I'd recommend..."

Implementation Pattern

The Memory-Augmented Prompt Builder

from typing import Callable, Optional
from dataclasses import dataclass

@dataclass
class PromptConfig:
    system_prompt: str
    memory_budget_tokens: int = 3000
    max_context_memories: int = 10
    include_working_memory: bool = True
    include_session_summary: bool = True
    include_long_term: bool = True

class MemoryAugmentedPrompt:
    def __init__(self, config: PromptConfig, memory: UnifiedMemory):
        self.config = config
        self.memory = memory

    async def build_prompt(
        self,
        user_message: str,
        session_id: str,
        custom_instructions: Optional[str] = None
    ) -> list[dict]:
        """Build a prompt with injected memory context."""
        messages = []

        # System prompt
        system_content = self.config.system_prompt
        if custom_instructions:
            system_content += f"\n\n{custom_instructions}"

        # Recall relevant memories
        memory_context = await self._recall_context(
            user_message, session_id
        )

        if memory_context:
            system_content += f"\n\n{memory_context}"

        messages.append({"role": "system", "content": system_content})

        # Add conversation history from working memory
        if self.config.include_working_memory:
            working_history = self.memory.working.get_context()
            messages.extend(working_history)

        # Add current user message
        messages.append({"role": "user", "content": user_message})

        return messages

    async def _recall_context(self, query: str, session_id: str) -> str:
        """Recall and format relevant memory context."""
        context_parts = []

        # Session summary (if available)
        if self.config.include_session_summary:
            summary = self.memory.short_term.get_summary()
            if summary:
                context_parts.append(f"Current session summary: {summary}")

        # Recent turns
        recent_turns = self.memory.short_term.get_recent_turns(5)
        if recent_turns:
            turn_text = "\n".join([
                f"- {t['role']}: {t['content'][:100]}..."
                for t in recent_turns
            ])
            context_parts.append(f"Recent conversation:\n{turn_text}")

        # Relevant long-term memories
        if self.config.include_long_term:
            long_term = self.memory.long_term.recall(query, top_k=5)
            if long_term:
                memories_text = "\n".join([
                    f"- [{m.type}] {m.content}"
                    for m in long_term
                ])
                context_parts.append(f"Relevant memories:\n{memories_text}")

        return "\n\n".join(context_parts)

Dynamic Prompt Templates

class DynamicPromptTemplate:
    def __init__(self, template: str, memory: UnifiedMemory):
        self.template = template
        self.memory = memory

    async def render(
        self,
        variables: dict,
        query: str
    ) -> str:
        """Render template with memory-injected variables."""
        # Base variables
        rendered = self.template.format(**variables)

        # Inject memory if template has placeholder
        if "{memory_context}" in rendered:
            memory_ctx = await self._build_memory_context(query)
            rendered = rendered.replace("{memory_context}", memory_ctx)

        # Inject user preferences if available
        if "{user_preferences}" in rendered:
            prefs = await self._get_user_preferences()
            rendered = rendered.replace("{user_preferences}", prefs)

        return rendered

    async def _build_memory_context(self, query: str) -> str:
        memories = self.memory.recall(query, max_tokens=2000)
        if not memories:
            return "No relevant context available."

        return "Relevant context from past interactions:\n" + \
               "\n".join([f"- {m.content}" for m in memories])

    async def _get_user_preferences(self) -> str:
        prefs = self.memory.long_term.recall(
            "user preferences and settings",
            top_k=5
        )
        if not prefs:
            return "No user preferences stored."

        return "Known user preferences:\n" + \
               "\n".join([f"- {p.content}" for p in prefs])

Prompt Patterns for Memory Injection

Pattern 1: Context Injection

Add memory as a context block in the system prompt:

system_prompt = f"""
You are a helpful coding assistant.

{memory_context_block}

Current task: {user_query}
"""

Pattern 2: Few-Shot Examples from Memory

Use past successful interactions as few-shot examples:

async def build_few_shot_prompt(query, memory):
    # Find similar past interactions that succeeded
    similar = memory.long_term.recall(
        query,
        filter={"outcome": "success"},
        top_k=3
    )

    examples = "\n".join([
        f"User: {s.query}\nAssistant: {s.response}"
        for s in similar
    ])

    return f"""
You are a helpful assistant. Here are examples of successful responses:

{examples}

Now respond to: {query}
"""

Pattern 3: Instruction Refinement

Use stored preferences to refine instructions:

async def build_refined_prompt(base_instructions, memory):
    # Get user-specific refinements
    refinements = memory.long_term.recall(
        "instructions, preferences, corrections",
        top_k=5
    )

    refinement_text = "\n".join([
        f"- {r.content}" for r in refinements
    ])

    return f"""
{base_instructions}

User-specific refinements based on past interactions:
{refinement_text}
"""

Pattern 4: Self-Referential Memory

The agent can query and modify its own memory:

MEMORY_TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search_memory",
            "description": "Search past memories for relevant information",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "What to search for"}
                },
                "required": ["query"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "store_memory",
            "description": "Store a new memory for future reference",
            "parameters": {
                "type": "object",
                "properties": {
                    "content": {"type": "string", "description": "What to remember"},
                    "type": {"type": "string", "enum": ["fact", "decision", "preference"]}
                },
                "required": ["content", "type"]
            }
        }
    }
]

Advanced: Contextual Memory Selection

Select different memory strategies based on the query type:

class ContextualMemorySelector:
    def __init__(self, memory):
        self.memory = memory
        self.strategies = {
            "factual": self._factual_strategy,
            "conversational": self._conversational_strategy,
            "task_oriented": self._task_strategy,
        }

    async def select_memories(self, query: str) -> list[Memory]:
        query_type = self._classify_query(query)
        strategy = self.strategies.get(query_type, self._default_strategy)
        return await strategy(query)

    def _classify_query(self, query: str) -> str:
        if any(word in query.lower() for word in ["what", "when", "where", "who"]):
            return "factual"
        elif any(word in query.lower() for word in ["let's", "continue", "we were"]):
            return "conversational"
        else:
            return "task_oriented"

    async def _factual_strategy(self, query: str) -> list[Memory]:
        # Prioritize long-term facts
        return self.memory.long_term.recall(query, top_k=10)

    async def _conversational_strategy(self, query: str) -> list[Memory]:
        # Prioritize recent context
        recent = self.memory.short_term.get_recent_turns(20)
        summary = self.memory.medium_term.recall_relevant(query, 3)
        return recent + summary

    async def _task_strategy(self, query: str) -> list[Memory]:
        # Balanced across all layers
        return self.memory.recall(query, max_tokens=3000)

Best Practices

1. Be transparent about memory — Tell users when you’re using past context. “Based on our previous conversation about X…” builds trust.

2. Respect memory boundaries — Don’t inject sensitive information from past sessions unless explicitly relevant. Privacy matters.

3. Allow memory override — Let users opt out of memory injection for specific queries. “Forget context” should be a valid request.

4. Monitor injection quality — Track whether injected memories actually help. If agents perform worse with memory, the injection strategy needs tuning.

5. Version your prompt templates — As you refine memory injection patterns, version your templates to track what works.

Conclusion

Memory-augmented prompts transform static agents into context-aware assistants. By automatically injecting relevant past context into every prompt, agents maintain continuity without requiring users to repeat themselves. The key is building a memory system that selects the right information for each query, respecting both token budgets and user privacy.

References:

• CrewAI Memory and Context
• OpenAI Agents SDK Context Management