{
  "id": 15726852,
  "title": "When an Agent Changes One Field, How Do We Know It Got It Right?",
  "url": "https://urgent.news/2026/10/20/when-an-agent-changes-one-field-how-do-we-know-it-got-it-right",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-20T09:47:28.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/yuki-uix/when-an-agent-changes-one-field-how-do-we-know-it-got-it-right-h6f"
  },
  "original_language": "en",
  "account": "As I delved deeper into Agent interactions, I pondered a crucial question: if I treated an Agent as a user of a project, could the existing documentation, APIs, and permissions enable it to successfully complete a rule configuration task? To investigate, I selected a specific rule change. My objective was to modify a description for a single target rule, while leaving the rest of the configuration unchanged. I aimed to observe the server-side checks and present the candidate and check results to the user for review, without saving or activating the changes.\n\nInitially, I anticipated that a straightforward pass or fail assessment from an Agent would suffice to determine if the task was executed correctly. However, I soon realized the need to establish clear acceptance criteria for what constituted a successful outcome. This article outlines the steps I took to define those criteria.\n\nThe first step was to determine the input given to the Agent. I discovered that there were multiple ways to phrase the same request, such as \"Bring me the file for the next meeting\" or \"Get the specified material from the seventh cabinet in the second row beside table three.\" These varying inputs could influence the Agent's approach. To ensure a fair assessment, I needed to fix the conditions I provided to the Agent. This meant conducting a fresh conversation, without any prior debugging discussions or the expected answer, to avoid treating subsequent executions as independent completions.\n\nNext, I had to consider the evidence that would validate the Agent's actions. I realized that using the same change through different routes could yield distinct results. By comparing the evidence from each route, I aimed to establish a consistent definition of \"success.\" In one route, the Agent performed a full candidate validation, including hot-reload checks and tracking both current and candidate ETags. In the second route, it applied the recipe change to the full configuration in memory and validated it, but without generating a candidate ETag or hot-reload check. The second route did not return a current_etag tied to the baseline configuration.\n\nThe third question arose: does a successful API response indicate that only the target field was modified? I discovered that configuration operations supported a merge mode, allowing multiple fields to be updated in a single request. If the Agent successfully changed the target description while inadvertently deleting other fields, my current checks might still consider the task a success. To ensure accuracy, I examined the implementation used in this experiment and found that merge mode replaced the entire list for fields such as decisions, rather than updating individual items. This meant that, in this case, changing the target.description correctly could unintentionally remove other rules present in the original configuration.\n\nIn conclusion, while the API responses provided valuable evidence of the Agent's actions, they did not always guarantee that only the intended changes were made. To ensure the correctness of the Agent's work, further investigation and additional checks would be necessary to confirm that the target field was the only modification.",
  "summary": "As I found myself spending more time in Agent conversations than on web pages when I opened my laptop, I started to wonder: if I treated an Agent as a user of a project, could the existing documentation, APIs, and permissions support it in completing an operation correctly? For this experiment, I chose a rule configuration task. The task was small: prepare a description change for one target…",
  "key_points": [
    "Agent treated as user to test rule configuration task",
    "Defined acceptance criteria for successful outcome",
    "API responses don't guarantee only target field modified"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}