Agent Change Management: Peer Review, Staging, and Rollback for Prompt Updates

Changing a prompt in production is like deploying code without a test suite — it might work, or it might silently break your agent’s behavior. Agent change management brings software engineering discipline to prompt and configuration updates: peer review, staging environments, automated evals, and safe rollbacks.

The Problem with Ad-Hoc Prompt Changes

Without change management:

• A developer tweaks a system prompt to fix one edge case
• The fix works for that case but degrades performance elsewhere
• No one notices until customers complain days later
• Rolling back means frantically searching Slack for “what was the old prompt?”

Change management prevents this by treating agent configurations as code.

Version Control for Agent Configs

Structure Your Agent Config

# agent-config.yaml
version: "1.2.3"
model: "gpt-4o"
temperature: 0.7

system_prompt: |
  You are a customer support agent for Acme Corp.
  
  Guidelines:
  - Be helpful and concise
  - Always verify the customer's identity before sharing account details
  - Escalate billing disputes to a human agent
  
  Current date: {{current_date}}

tools:
  - name: lookup_order
    description: Look up an order by ID
    parameters:
      order_id:
        type: string
        description: The order ID
  
  - name: process_refund
    description: Process a refund for an order
    parameters:
      order_id:
        type: string
      reason:
        type: string

guardrails:
  max_tokens: 2000
  blocked_patterns:
    - "ignore previous"
    - "system prompt"
  required_disclaimers:
    - "Terms and conditions apply"

Git-Based Version Control

# Each change is a commit
git add agent-config.yaml
git commit -m "feat: add refund processing tool"

# Tags for releases
git tag v1.2.3
git tag -a v1.3.0 -m "Add order lookup tool"

# Branch for experiments
git checkout -b experiment/temperature-tuning

The Change Pipeline

Step 1: Propose Changes

class AgentChangeProposal:
    def __init__(self):
        self.proposals = []
    
    def create_proposal(
        self,
        author: str,
        current_version: str,
        changes: dict,
        rationale: str,
        eval_criteria: list[str]
    ) -> dict:
        proposal = {
            "id": f"proposal-{len(self.proposals) + 1}",
            "author": author,
            "current_version": current_version,
            "changes": changes,
            "rationale": rationale,
            "eval_criteria": eval_criteria,
            "status": "draft",
            "created_at": datetime.now().isoformat(),
        }
        self.proposals.append(proposal)
        return proposal

Step 2: Automated Evaluation

Run the proposed config against your eval suite:

class EvalRunner:
    def __init__(self, eval_suite: list[dict]):
        self.eval_suite = eval_suite
    
    async def run_evals(
        self,
        config: dict,
        baseline_config: dict = None
    ) -> dict:
        results = {"pass": 0, "fail": 0, "regressions": []}
        
        for eval_case in self.eval_suite:
            # Run with proposed config
            proposed_result = await self._run_eval(config, eval_case)
            
            # Run with baseline for comparison
            if baseline_config:
                baseline_result = await self._run_eval(baseline_config, eval_case)
                
                # Check for regressions
                if self._is_regression(baseline_result, proposed_result):
                    results["regressions"].append({
                        "eval": eval_case["name"],
                        "baseline_score": baseline_result["score"],
                        "proposed_score": proposed_result["score"],
                    })
            
            if proposed_result["passed"]:
                results["pass"] += 1
            else:
                results["fail"] += 1
        
        results["pass_rate"] = results["pass"] / len(self.eval_suite)
        results["regression_count"] = len(results["regressions"])
        
        return results
    
    def _is_regression(self, baseline: dict, proposed: dict) -> bool:
        return proposed["score"] < baseline["score"] * 0.95  # 5% threshold

Step 3: Peer Review

class ReviewSystem:
    def request_review(self, proposal_id: str, reviewers: list[str]):
        proposal = self._get_proposal(proposal_id)
        proposal["status"] = "review"
        proposal["reviewers"] = reviewers
        proposal["reviews"] = {}
        
        # Notify reviewers
        for reviewer in reviewers:
            self._send_notification(reviewer, proposal)
    
    def submit_review(
        self,
        proposal_id: str,
        reviewer: str,
        approved: bool,
        comments: str
    ):
        proposal = self._get_proposal(proposal_id)
        proposal["reviews"][reviewer] = {
            "approved": approved,
            "comments": comments,
            "timestamp": datetime.now().isoformat()
        }
        
        # Check if all reviewers have approved
        all_approved = all(
            r["approved"] for r in proposal["reviews"].values()
        )
        
        if all_approved:
            proposal["status"] = "approved"
            self._trigger_deployment(proposal_id)
        elif any(not r["approved"] for r in proposal["reviews"].values()):
            proposal["status"] = "changes_requested"

Step 4: Staging Deployment

class StagingEnvironment:
    def __init__(self):
        self.shadow_traffic_percentage = 0.1  # 10% of traffic
    
    async def deploy_to_staging(self, config: dict) -> str:
        # Deploy alongside production
        staging_endpoint = await self._provision_staging(config)
        
        # Route a percentage of traffic to staging
        await self._configure_traffic_split(
            production=self.production_endpoint,
            staging=staging_endpoint,
            split_ratio=self.shadow_traffic_percentage
        )
        
        # Monitor for issues
        monitor = StagingMonitor(staging_endpoint)
        await monitor.watch(duration_hours=24)
        
        return staging_endpoint
    
    async def promote_to_production(self, config: dict):
        # Validate staging results
        staging_results = await self._get_staging_results()
        
        if staging_results["error_rate"] > 0.01:  # 1% threshold
            raise PromotionError("Staging error rate too high")
        
        if staging_results["latency_p99"] > self.latency_budget:
            raise PromotionError("Staging latency exceeds budget")
        
        # Deploy to production
        await self._deploy_production(config)

Step 5: Safe Rollback

class RollbackManager:
    def __init__(self):
        self.version_history = []
    
    def deploy_version(self, config: dict, version: str):
        # Save current version for rollback
        current = self._get_current_config()
        self.version_history.append({
            "version": version,
            "config": current,
            "deployed_at": datetime.now().isoformat()
        })
        
        # Deploy new version
        self._apply_config(config)
    
    def rollback(self, target_version: str = None):
        if target_version:
            version = self._find_version(target_version)
        else:
            version = self.version_history[-1]
        
        self._apply_config(version["config"])
        
        # Log rollback
        logger.warning(
            f"Rolled back from {self._current_version()} "
            f"to {version['version']}"
        )
    
    def rollback_if_degraded(
        self,
        metric_name: str,
        threshold: float,
        check_interval: int = 60
    ):
        """Automatic rollback based on metrics."""
        while True:
            current_value = self._get_metric(metric_name)
            
            if current_value > threshold:
                logger.error(
                    f"Metric {metric_name} exceeded threshold: "
                    f"{current_value} > {threshold}. Rolling back."
                )
                self.rollback()
                break
            
            time.sleep(check_interval)

CI/CD Integration

GitHub Actions Workflow

name: Agent Config Deployment

on:
  push:
    branches: [main]
    paths: ['agent-config.yaml']

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run evals
        run: python run_evals.py --config agent-config.yaml
      
      - name: Check eval results
        run: |
          python check_eval_results.py \
            --min-pass-rate 0.95 \
            --max-regressions 0

  staging:
    needs: eval
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to staging
        run: python deploy.py --env staging
      
      - name: Run smoke tests
        run: python smoke_tests.py --env staging
      
      - name: Monitor staging (1 hour)
        run: python monitor.py --env staging --duration 3600

  production:
    needs: staging
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: python deploy.py --env production
      
      - name: Verify production
        run: python verify.py --env production --duration 1800
      
      - name: Auto-rollback on issues
        if: failure()
        run: python rollback.py --env production

Monitoring and Alerting

class AgentHealthMonitor:
    def __init__(self):
        self.baselines = {}
    
    def check_health(self, config_version: str) -> dict:
        metrics = self._collect_metrics()
        
        alerts = []
        baseline = self.baselines.get("stable")
        
        if baseline:
            # Check for degradation
            if metrics["success_rate"] < baseline["success_rate"] * 0.95:
                alerts.append({
                    "type": "degradation",
                    "metric": "success_rate",
                    "current": metrics["success_rate"],
                    "baseline": baseline["success_rate"],
                    "severity": "high"
                })
            
            if metrics["avg_latency"] > baseline["avg_latency"] * 1.5:
                alerts.append({
                    "type": "latency_spike",
                    "metric": "avg_latency",
                    "current": metrics["avg_latency"],
                    "baseline": baseline["avg_latency"],
                    "severity": "medium"
                })
        
        return {
            "healthy": len(alerts) == 0,
            "alerts": alerts,
            "metrics": metrics
        }

Best Practices

• Every config change goes through the pipeline — No exceptions, no hotfixes
• Keep eval suites updated — Add new test cases for each bug fix
• Monitor after every deployment — At least 1 hour of heightened monitoring
• Document rollbacks — Record why each rollback happened for future prevention
• Use feature flags — For gradual rollouts and quick toggles

Conclusion

Agent change management brings engineering discipline to prompt and configuration updates. By versioning configs, running automated evals, requiring peer review, deploying to staging, and monitoring production, you catch issues before they affect users. When issues do slip through, automated rollback gets you back to a known-good state in minutes.