Feedback Loops: Turning Human Edits into Agent Improvements

Every time a human rewrites your agent's output, you throw away a training signal. A feedback loop turns those edits into datasets, evals, and measurable model improvements.

Published on • October 13, 2026

AI Assistant

The most valuable data your AI product generates is not the user’s prompt. It is the edit the user makes after your agent answers. Someone retyped your draft, deleted a paragraph, changed a number, or rewrote the tone — and then that revision vanished into a messages table nobody queries.

Teams that ship agents well treat every human edit as a free label. This post is about building the loop: capture edits as structured preference data, turn them into evals first and training data second, and close the loop so the agent measurably improves.

Step 1: Capture the edit, not just the final text

The single biggest mistake is storing only the post-edit output. Without the model’s original draft, there is no comparison — and comparison is the whole game. Log the triple:

{
  "task": "summarize_ticket",
  "prompt": "...",
  "model_output": "Draft the agent produced...",
  "human_output": "What the human saved...",
  "edit_distance": 0.42,
  "accepted": false,
  "timestamp": "2026-10-13T04:00:00Z"
}

Richer signals are worth the effort when you can get them:

Inline diffs — which spans the human touched, and whether they added or deleted.
Time-to-edit — a rewrite in four seconds is a rejection; two minutes of light touch-ups is an accept-with-nits.
Explicit affordances — thumbs up/down with a reason enum (“too long”, “wrong format”, “factual error”) beats inference.
Context — the retrieved documents the agent saw, so you can tell “bad reasoning” from “bad retrieval.”

Store this in your product database and export it regularly to a dataset. Hugging Face Datasets is the standard home for that export: Apache Arrow-backed, versionable, and push_to_hub gives you a diffable, shareable history that your training jobs and your eval harness can both consume.

from datasets import Dataset

ds = Dataset.from_list(edit_records)
ds.push_to_hub("acme/agent-edits-2026q3", private=True)

Step 2: Separate the two consumers of feedback

Human edits feed two very different processes, and conflating them is how teams stall:

Evals — turn edits into tests immediately, while the loop is hot. Every accepted rewrite becomes a case: prompt → expected characteristics. Even a lightweight regression suite (“the agent must not exceed three bullets”, “must cite the retrieved source”) built from last week’s edits catches drift on the next deploy. This pays off within days.

Training data — the slower path, but the one that compounds. Preference pairs — (draft, edited) — are exactly the format post-training libraries expect.

Step 3: From edits to training pairs

Hugging Face’s TRL library covers the modern post-training menu: SFTTrainer for supervised fine-tuning, DPOTrainer and KTOTrainer for offline preference optimization, ORPOTrainer, plus online methods like GRPOTrainer. The mapping from your logs:

(prompt, edited_output) pairs → SFT. The human’s rewrite is the target.
(prompt, draft, edited) triples → DPO, where the draft becomes the rejected completion and the edit the chosen one. No reward model needed.
Only implicit signals (accept/reject clicks, no rewrite) → KTO, which is designed for binary feedback rather than preference pairs.

The DPO path is where edits shine: a human rewrite is literally a ranked pair — “this, not that” — and DPO learns directly from the comparison instead of from a scalar reward a model had to guess at.

Quality gates matter more than quantity here. Filter out trivial edits (typo fixes, whitespace), dedupe near-identical prompts, and stratify by task type so one noisy workflow does not dominate the distribution. A thousand clean pairs from your highest-traffic task beats a hundred thousand scraped across the long tail.

Step 4: Close the loop

The loop is not closed when training finishes. It closes when you can answer “did it get better?”:

Hold out an eval set from the same period as the training data, before training starts.
Run the candidate against the holdout and against the deployed model on identical inputs.
Ship behind a flag — route a slice of traffic to the candidate.
Watch the same metrics you captured in step one: edit rate, edit distance, time-to-edit. If the agent improved, the human edit distance should drop.
Promote or roll back, and keep logging — the next batch of edits is already accumulating.

That last point is the mindset shift: feedback capture never stops. The model you deploy today is the generator of tomorrow’s training data.

Practical pitfalls

Selection bias. The humans who edit most are the pickiest. Sample across users and tasks, or weight deliberately.
Privacy and consent. User-edited text may contain customer data. Get contractual rights to use it for training, anonymize, and keep the dataset private.
Feedback loops that amplify errors. If the agent is confidently wrong in a way users rarely catch, the dataset quietly teaches more of the same. Pair edit-driven training with adversarial evals and domain-expert review on a sample.
Rewarding verbosity. If humans tend to lengthen outputs, naive SFT learns to ramble. Normalize or explicitly penalize in your eval.

Where to start this week

You do not need a training pipeline to benefit. Minimum viable loop: log (prompt, draft, edited) triples to a table, export weekly to a Datasets repo, and write five regression tests from the worst edits you see. That alone changes your agent more than most fine-tuning experiments, and it positions you for DPO the day the volume justifies it.