The Rehearsal Layer — Or, an Escrow You Have Never Opened Is Indistinguishable from No Escrow
Yesterday's piece ended with a test: four items, PASS / FAIL / UNKNOWN, UNKNOWN counts as FAIL. It was a good test as far as it went. It had a hole, and the hole is the whole subject of today. The test asks whether the box contains what a successor needs. It does not ask whether anyone has ever opened it. Those are different questions, and only one of them is answerable from the outside. Two…
The idea that an escrow, which is never opened, is indistinguishable from one that does not exist is explored in this report. Two companies, on the day before they need continuity, both have an escrow but are essentially the same from an outside perspective. The key difference only becomes apparent when it matters most - the day of actual need.
The purpose of a rehearsal is to uncover these differences in a controlled environment during a drill. However, the economics of conducting such drills are often seen as poor - they require resources now for uncertain future benefits. The cost is certain, while the benefit is probabilistic and may not materialize.
There are several reasons why teams don't rehearse. The main one is that the perceived cost outweighs the uncertain benefit. A drill involves significant effort with no immediate tangible reward. The benefit realized from a successful drill is only realized if an incident occurs, which may never happen. Even then, the positive impact might not be easily identifiable.
However, the author argues that these drills are crucial. They help to identify discrepancies between the documented processes and the actual implementation. These drills can reveal issues like expired tokens, outdated consumers, or missing approvers. This information is invaluable as it highlights weaknesses in the recovery plan.
There are four levels of drills, each increasing in complexity and discomfort. The drills progress from simply opening the escrow to rotating credentials, rotating credentials in a real environment, and finally having someone unfamiliar with the system perform the drill. The goal is to uncover the weaknesses in the system and the processes surrounding it.
In conclusion, the value of these drills lies in the insights they provide. The small investment in time and resources can yield significant benefits in terms of preparedness and resilience. The drills should be conducted regularly - at least twice a year - and the results should be documented meticulously. This way, the organization can continuously improve its recovery plan and be better equipped to handle potential incidents.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
