Datadog Builds a Specialist AI Model for SRE Investigations

Datadog has introduced Scout, a specialized AI model designed to investigate production problems as part of its Bits Investigation AI SRE product. The model can analyze an alert, examine telemetry data, and suggest potential causes of the issue. Currently, Datadog engineers and a limited number of customers are testing Scout in a closed preview.

The primary focus of Scout's development was cost efficiency. Investigating issues using an investigation tool can involve numerous model calls, as the system tries queries, evaluates results, and decides on the next steps to take. General-purpose models can perform these tasks, but Datadog argues that the costs associated with running them on every alert make it impractical.

The aim is for Scout to serve as an always-on first responder, gathering evidence and identifying likely causes before an engineer becomes involved. According to the AlertEval benchmark, Scout medium achieved a score similar to GPT-6.1 Sol high and Claude Opus 5.5 high, while its estimated inference cost was between three and eight times lower than these models. At the xhigh setting, Scout matched Claude Fable 5.1 xhigh's performance at just one quarter of the cost.

The model, based on GLM-5.3 Flash, was post-trained using reinforcement learning with incident simulations created from historical patterns. The simulations included an alert, queryable telemetry, and a known cause, but did not contain any customer data. The training process rewarded Scout for finding the underlying cause and supporting its conclusions with evidence, while discouraging it from making excessive claims or stopping investigation prematurely.

Datadog conducted these evaluations internally, including the tests on newer comparison models, so independent results would be valuable. Scout has been implemented in observability platforms as companies work out the optimal balance of incident handling between agents and the AI model.

Several other vendors have been developing their approaches to SRE, similar to Datadog's Scout. AWS DevOps Agent combines telemetry from various monitoring products with source code, deployment records, and runbooks, while Honeycomb's Canvas Agent acts more like a manual investigation tool, allowing engineers to capture familiar debugging procedures as they work.

Splunk and New Relic have gone further in the repair stage, with Splunk AI SRE generating a probable cause with an evidence chain and a connected Claude agent preparing code changes in a sandbox.