{
  "id": 20465899,
  "title": "Chaos engineering with Dynatrace: measure before you inject a failure",
  "url": "https://urgent.news/2026/11/17/chaos-engineering-with-dynatrace-measure-before-you-inject-a-failure",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-17T17:30:19.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/spanobserver/chaos-engineering-with-dynatrace-measure-before-you-inject-a-failure-3348"
  },
  "original_language": "en",
  "account": "In a recent article, Dynatrace shares their approach to chaos engineering, which involves measuring before injecting failures in production. The primary question to consider is whether one can determine within ten minutes if an experiment was successful or if it should be stopped. The article emphasizes that a chaos experiment should be treated as a hypothesis test, with a clear contract outlining what will be broken, what is expected, and the threshold that will trigger an immediate stop.\n\nTo run a successful experiment, start by establishing a baseline over a normal period, such as seven days, ensuring the traffic level is comparable to the experiment. Record key metrics, including failure rates and response times, as well as the number of open problems and their usual duration. If the baseline is already poor, it is advisable to postpone the experiment.\n\nNext, create two platform building blocks to turn the hypothesis into checkable objectives: a Service Level Objective (SLO) and an Service Reliability Guardian (SRG) guardian. The SLO measures an SLI, which is a series normalised between 0 and 100% computed using a DQL query, compared against a target over an evaluation period. The error budget, the gap between the status and the target, helps determine how fast the error budget is being spent. The SRG guardian groups objectives, each fed by a DQL query and judged against warning and failure thresholds, serving as the referee for the experiment.\n\nOnce the objectives are set, automate the verdict using a workflow. This workflow can call the guardian action after the injection window, using an event trigger or an on-demand call. The result will expose a validation status, such as pass, warning, fail, or error, along with an identifier and a link to the validation. Decision-making, such as widening the blast radius or stopping the experiment, is based on this validation status.\n\nIn conclusion, chaos engineering with Dynatrace begins with measurement—setting up a baseline, stop thresholds, and an automated verdict. Only then can the failure be introduced.",
  "summary": "Before scheduling a chaos engineering with Dynatrace experiment in production, one question matters: « if I slow down or cut this component, how will I know within ten minutes that the experiment worked, or that it must stop? » Without a numeric answer, a deliberate failure is just a failure. This article starts from that operations decision. It does not describe a fault-injection tool: in the…",
  "key_points": [
    "Dynatrace measures before injecting failures in production",
    "Chaos experiment treated as hypothesis test with clear contract",
    "SLO and SRG guardian automate verdict for experiment decision"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}