Urgent.News

What's breaking now, across hundreds of outlets.

AI

How an AI Support Agent Learns From Agent Edits

Every time a support agent edits an AI draft before sending it, they show the system how the company actually handles that case. In the 30 days to 19 November 2026, agents in the deployment from our case study edited 19,339 AI drafts. That is the most valuable training material the system has, and it is easy to lose. This is the loop we built to use it, and what it found. Keep both versions The…

When a support representative modifies an AI-generated response before dispatch, they inform the system of how the actual company addresses the issue. Over the 30 days leading up to November 19, 2026, agents in the case study deployment made 19,339 edits to AI drafts. This serves as the system's most valuable training material, though it can be easily overlooked. The following process utilizes this valuable feedback, detailing how it was implemented and the results observed.

The system requires two versions of each response: the AI's initial draft and the final version sent by the agent. Unlike many other systems that overwrite the draft with the edited version, this method stores both texts separately. Before making any comparisons, it is crucial to test the system on replies generated without human intervention. The texts must match in these cases to ensure the comparison method is functioning correctly.

To extract insights from the edited replies, the system follows five steps. First, it identifies the differences between the AI's draft and the final version. For each edited reply, these changes are categorized into themes. Next, the system proposes a new rule if a theme appears frequently and independently across multiple edited replies. This rule is then verified by replaying real past cases with and without the proposed rule, measuring the gap between the two approaches, and diagnosing any discrepancies.

In the initial run, conducted over seven days with 300 complex after-sales replies, six out of eight practices that passed the threshold were already covered by existing rules. The remaining two practices revealed that the system, despite having been instructed on specific scenarios, did not execute them consistently. The root cause was found to be the presence of the same matter described in multiple locations with slight variations.

For instance, when a customer's fault repair was handled at a local shop, agents would frequently add the reimbursement limit to drafts even if it was omitted. This issue stemmed from five different rules describing the limit in varying ways across the reply principles, constraint entries, solution descriptions, and procedure documents.

The solution was to standardize the wording of the limit across all five rule locations. Once this standardization was implemented, the drafts consistently included the limit when applicable.

When a knowledge base rule, an existing constraint, or a procedure document contradicts the consistent actions of agents, the agents' practices take precedence, and the document is updated accordingly. However, for a rule change to occur, the practice must be consistently observed across several agents and numerous replies. A single agent's isolated choice or an erroneous reply does not warrant a rule update.

The distilled rules stored in the experience library should be written in a way that makes sense without reference to the original ticket, including essential details such as situational contexts, actions to be taken, and exceptions. Duplicate entries are merged to avoid redundancy and conflicting information. As part of the distillation process, any entries containing contradictory information with the current policy are discarded.

All new draft submissions must pass through a distillation step to ensure they align with the distilled rules.

To validate a proposed change, the system compares it against real past tickets in a replay scenario. Two key factors determine the relevance of the test: freezing the data that can change over time and running each case twice to account for potential variations in model outputs. If a proposed change is not clearly better across both runs, it will not be deployed. This step helps ensure that only effective changes are implemented.

Upon reviewing a sample of 400 edited drafts, it was found that 53 percent of the edits involved removing promises agents did not want to make, 22 percent added stock status information the draft had omitted, and 10 percent replaced generic statements like "we have no information" with specific answers drawn from the knowledge base. These three specific pieces of work highlight the types of changes agents frequently make to AI-generated responses.

This loop is designed for teams that regularly review AI output, requiring a substantial volume of edited replies, typically thousands per month, to effectively identify and distill consistent practices. The full details of how to implement this loop within your organization can be found in the free plan provided on ai.dingboai.com.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in AI

Building a WhatsApp AI assistant for Moroccan businesses (Darija) with Next.js, Supabase and Claude

Most small businesses in Morocco take orders on WhatsApp. The owner reads every message, types the menu again and again, and misses messages at night.

  • AI assistant answers customer inquiries in Moroccan Darija, French, or Arabic.
  • System uses single codebase for multiple merchants, isolating data via Supabase Postgres.
  • Reporter sells packaged Next.js, TypeScript, Supabase solution for developers and agencies.

Human-in-the-Loop AI Agents: Who Approves What

"Human in the loop" is easy to say and easy to get wrong. Put the person on every step and the AI saves nobody any time. Put the person nowhere and the first wrong refund lands on your bank statement.

  • AI handles product facts, policies, troubleshooting, and order lookups
  • Humans approve money-related decisions, reshipments, returns, and policy breaches
  • Approval loop ensures quick decision-making and human oversight

How AI Can Turn Hundreds of PDF Documents Into Structured Product Data

Many companies work with large amounts of product-related information every day. Technical datasheets, safety documents, certificates, bills of materials and other files often contain important…

  • AI can extract structured product data from hundreds of PDF documents
  • Manual document review becomes difficult with increasing products and documents
  • AI systems organize extracted data into categories for databases or systems

More from Sunday 22 November →