Chaos engineering with Dynatrace: measure before you inject a failure
Before scheduling a chaos engineering with Dynatrace experiment in production, one question matters: « if I slow down or cut this component, how will I know within ten minutes that the experiment worked, or that it must stop? » Without a numeric answer, a deliberate failure is just a failure. This article starts from that operations decision. It does not describe a fault-injection tool: in the…
In a recent article, Dynatrace shares their approach to chaos engineering, which involves measuring before injecting failures in production. The primary question to consider is whether one can determine within ten minutes if an experiment was successful or if it should be stopped. The article emphasizes that a chaos experiment should be treated as a hypothesis test, with a clear contract outlining what will be broken, what is expected, and the threshold that will trigger an immediate stop.
To run a successful experiment, start by establishing a baseline over a normal period, such as seven days, ensuring the traffic level is comparable to the experiment. Record key metrics, including failure rates and response times, as well as the number of open problems and their usual duration. If the baseline is already poor, it is advisable to postpone the experiment.
Next, create two platform building blocks to turn the hypothesis into checkable objectives: a Service Level Objective (SLO) and an Service Reliability Guardian (SRG) guardian. The SLO measures an SLI, which is a series normalised between 0 and 100% computed using a DQL query, compared against a target over an evaluation period. The error budget, the gap between the status and the target, helps determine how fast the error budget is being spent.
The SRG guardian groups objectives, each fed by a DQL query and judged against warning and failure thresholds, serving as the referee for the experiment.
Once the objectives are set, automate the verdict using a workflow. This workflow can call the guardian action after the injection window, using an event trigger or an on-demand call. The result will expose a validation status, such as pass, warning, fail, or error, along with an identifier and a link to the validation. Decision-making, such as widening the blast radius or stopping the experiment, is based on this validation status.
In conclusion, chaos engineering with Dynatrace begins with measurement—setting up a baseline, stop thresholds, and an automated verdict. Only then can the failure be introduced.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.