How to Check an Agent's Diagnosis Before It Touches Production

When teams prepare to add agents to their on-call workflows, a crucial decision lies in determining whether to let the agents act on production or simply provide advice. Causely's causal model now offers transparency into the reasoning behind the agent's claims, revealing which alternative diagnoses were considered, the evidence used to rule them out, and the potential impact if the chosen diagnosis was incorrect.

A recent discussion on kubernetes clarified that remediation agents should be divided into two components. The first component, an LLM, suggests potential actions like scaling, rolling back, cordoning, or draining. A separate deterministic layer then evaluates these suggestions against the current state of the live cluster before executing any action. The key insight from this conversation is that separating the proposal from the decision allows for a more rigorous review process.

One engineer proposes that agents should provide a rationale for the chosen diagnosis before executing any proposed action. This enables humans or policy engines to review the justification rather than merely accepting or rejecting the recommendation. Another engineer emphasizes the need for a veto mechanism that incorporates memory, such as blast-radius budgets per namespace, cooldown periods, and a rule to halt and alert a human if the expected effect of the action does not materialize within a certain timeframe.

Crucially, this veto must verify the actual state of the cluster rather than relying on the agent's self-reported state.

The primary challenge in this process is discerning when an agent's diagnostic capabilities are sufficient to permit action in production. Confidence in the agent's recommendation is not equivalent to accuracy. While a confidence score serves as an assertion, a list of potential explanations for the observed symptom, complete with the causal chain supporting each, is verifiable by a reviewer, either human or automated.

This addresses the underlying issue highlighted by the second engineer in the discussion: a veto that trusts the agent's self-report is merely another perspective, not a validation process.

To effectively evaluate an agent's diagnosis, one must ask "why this and not that?" Explanatory AI research suggests that explanations are more satisfying when they are contrastive - that is, they compare the chosen explanation to alternative possibilities that were not selected. This approach frames the explanation task as identifying the differences between the selected diagnosis (P) and other possible explanations (Q), rather than treating the comparison as an afterthought.

Causely's MCP server now provides two tools designed to facilitate this contrastive explanation process. The get_potential_diagnoses function returns all plausible diagnoses for a given symptom, along with the causal chain supporting each. Conversely, get_signal_potential_diagnoses analyzes an observed signal to identify potential causes and the causal connections between each candidate and the observed signal.

Both functions leverage the same underlying causal model that generates the primary diagnosis, presenting the reasoning in a more transparent manner.

Consider a scenario where a shared database connection pool experiences an increase in latency and timeouts affecting a checkout service. An agent solely relying on machine learning may diagnose a CPU spike on an adjacent service as the root cause and initiate a restart of that service to alleviate the issue. While this may provide temporary relief by clearing the CPU spike, it does not address the underlying problem of connection exhaustion on the shared database.

Subsequent incidents may trigger the same loop, as the agent repeatedly restarts the service without resolving the root cause.

By querying the causal model, Claude can generate an explanation that reveals the true root cause (connection exhaustion) and contrasts it with the alternative explanation (CPU spike). The causal chain for each candidate diagnosis is clearly presented, enabling a reviewer to assess whether the selected diagnosis accurately captures the underlying issue and whether acting on that diagnosis would be safe and effective.

Additionally, Claude can visualize the causal chain, providing a graphical representation of how each candidate diagnosis explains or fails to explain the observed symptom. This visualization aids in understanding the relationship between the potential causes and the observed signal, further enhancing the transparency of the explanation.

When evaluating the potential impact of an agent's diagnosis, it is essential to consider not only the blast radius of the proposed action but also the blast radius of the underlying problem that the diagnosis is trying to address. Diagnoses with small, well-understood blast radii present lower risks compared to those with large, uncertain blast radii, even if the agent's confidence level is similar.

The rank_entities function can help surface the entities most affected by a given diagnosis, allowing policy engines or on-call personnel to assess the potential impact and set appropriate blast-radius budgets accordingly.

The get_diagnosis_observable_signals tool can further assist in this evaluation by providing insights into the downstream signals that may be affected by a specific diagnosis. By understanding the observable signals associated with each candidate diagnosis, reviewers can better assess the potential consequences of choosing one explanation over another.

In summary, the key takeaway is that when assigning trust to an agent's diagnostic capabilities in production environments, it is crucial to review the reasoning behind the chosen diagnosis. Causally exposed explanations, contrastive explanations that compare the selected diagnosis to alternative possibilities, and mechanisms for assessing the blast radius of both the proposed action and the underlying problem are all essential components of a robust decision-making process.

By leveraging these tools and approaches, teams can ensure that agents are not only proposing safe actions but also providing explanations that can be independently verified and trusted before any changes are made to production systems.