{
  "id": 21268457,
  "title": "How an AI Support Agent Learns From Agent Edits",
  "url": "https://urgent.news/2026/11/22/how-an-ai-support-agent-learns-from-agent-edits",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-22T19:10:38.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/yzh317179958/how-an-ai-support-agent-learns-from-agent-edits-hhi"
  },
  "original_language": "en",
  "account": "When a support representative modifies an AI-generated response before dispatch, they inform the system of how the actual company addresses the issue. Over the 30 days leading up to November 19, 2026, agents in the case study deployment made 19,339 edits to AI drafts. This serves as the system's most valuable training material, though it can be easily overlooked. The following process utilizes this valuable feedback, detailing how it was implemented and the results observed.\n\nThe system requires two versions of each response: the AI's initial draft and the final version sent by the agent. Unlike many other systems that overwrite the draft with the edited version, this method stores both texts separately. Before making any comparisons, it is crucial to test the system on replies generated without human intervention. The texts must match in these cases to ensure the comparison method is functioning correctly.\n\nTo extract insights from the edited replies, the system follows five steps. First, it identifies the differences between the AI's draft and the final version. For each edited reply, these changes are categorized into themes. Next, the system proposes a new rule if a theme appears frequently and independently across multiple edited replies. This rule is then verified by replaying real past cases with and without the proposed rule, measuring the gap between the two approaches, and diagnosing any discrepancies.\n\nIn the initial run, conducted over seven days with 300 complex after-sales replies, six out of eight practices that passed the threshold were already covered by existing rules. The remaining two practices revealed that the system, despite having been instructed on specific scenarios, did not execute them consistently. The root cause was found to be the presence of the same matter described in multiple locations with slight variations. For instance, when a customer's fault repair was handled at a local shop, agents would frequently add the reimbursement limit to drafts even if it was omitted. This issue stemmed from five different rules describing the limit in varying ways across the reply principles, constraint entries, solution descriptions, and procedure documents. The solution was to standardize the wording of the limit across all five rule locations. Once this standardization was implemented, the drafts consistently included the limit when applicable.\n\nWhen a knowledge base rule, an existing constraint, or a procedure document contradicts the consistent actions of agents, the agents' practices take precedence, and the document is updated accordingly. However, for a rule change to occur, the practice must be consistently observed across several agents and numerous replies. A single agent's isolated choice or an erroneous reply does not warrant a rule update. The distilled rules stored in the experience library should be written in a way that makes sense without reference to the original ticket, including essential details such as situational contexts, actions to be taken, and exceptions. Duplicate entries are merged to avoid redundancy and conflicting information. As part of the distillation process, any entries containing contradictory information with the current policy are discarded. All new draft submissions must pass through a distillation step to ensure they align with the distilled rules.\n\nTo validate a proposed change, the system compares it against real past tickets in a replay scenario. Two key factors determine the relevance of the test: freezing the data that can change over time and running each case twice to account for potential variations in model outputs. If a proposed change is not clearly better across both runs, it will not be deployed. This step helps ensure that only effective changes are implemented.\n\nUpon reviewing a sample of 400 edited drafts, it was found that 53 percent of the edits involved removing promises agents did not want to make, 22 percent added stock status information the draft had omitted, and 10 percent replaced generic statements like \"we have no information\" with specific answers drawn from the knowledge base. These three specific pieces of work highlight the types of changes agents frequently make to AI-generated responses.\n\nThis loop is designed for teams that regularly review AI output, requiring a substantial volume of edited replies, typically thousands per month, to effectively identify and distill consistent practices. The full details of how to implement this loop within your organization can be found in the free plan provided on ai.dingboai.com.",
  "summary": "Every time a support agent edits an AI draft before sending it, they show the system how the company actually handles that case. In the 30 days to 19 November 2026, agents in the deployment from our case study edited 19,339 AI drafts. That is the most valuable training material the system has, and it is easy to lose. This is the loop we built to use it, and what it found. Keep both versions The…",
  "key_points": [
    "Agents edit AI drafts 19,339 times in 30 days",
    "System stores AI draft and final version separately",
    "Distilled rules guide consistent AI response generation"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}