{
  "id": 20430033,
  "title": "Datadog Brings Production Experimentation to AI Agent Observability",
  "url": "https://urgent.news/2026/11/17/datadog-brings-production-experimentation-to-ai-agent-observability",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-17T12:00:00.000Z",
  "source": {
    "name": "InfoQ",
    "slug": "infoq",
    "url": "https://www.infoq.com/news/2026/11/datadog-ai-agent-observability/"
  },
  "original_language": "en",
  "account": "Datadog enhances its Agent Observability platform with Online Experiments, enabling engineering teams to compare AI agent versions using real production traffic instead of solely relying on predefined evaluation datasets. The feature integrates agent evaluation scores with operational metrics like latency, token usage, execution cost, and errors, helping teams determine if model, prompt, retrieval, or tool changes truly improve the agent. This addresses the challenge where a change may perform better in controlled tests but behave differently in unpredictable user requests.\n\nDatadog combines production telemetry, execution traces, and controlled experimentation, offering evidence of both outcomes and behaviors. While offline evaluations are crucial, they are only as reliable as the datasets and scoring criteria. Online experiments test agent behavior against real user interactions, providing a more realistic performance view and exposing weaknesses unseen in test cases.\n\nTeams can compare control and modified variants by distributing production traffic randomly, giving a realistic performance assessment under actual workloads. Feature flags allow engineers to test changes on a small user subset before broader rollout. The approach connects evaluation scores with conventional operational metrics, enabling teams to measure response quality, accuracy, cost, and errors.\n\nFor instance, Datadog's experiment compared GPT-5.6 Luna and GPT-6 Luna, finding a 44.3% cost reduction with similar quality scores (3.84 and 3.86). However, operational results indicated potential issues like increased execution duration and errors. Inspecting traces revealed frequent invalid tool parameters, which were addressed before proceeding with the model change. This feedback loop between production behavior and development testing helps establish whether agents perform better in real conditions, becoming as critical as traditional software testing.",
  "summary": "Datadog is extending its Agent Observability platform with Online Experiments, allowing engineering teams to compare AI agent versions against real production traffic rather than relying exclusively on predefined evaluation datasets. By Craig Risi",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}