Recovery testing: restore order and verification, not just restore success

Restore success alone does not equate to recovery. A backup that restores cleanly confirms the data's readability, yet it fails to confirm whether the organization can swiftly return to operations. The complexity lies in the sequence services resume, the essential layers such as identity and network required first, and if the restored data is free from corruption.

Unfortunately, many teams gauge the incorrect metric. Recovery assessments that merely verify file counts or database checksums can pass even when the backup harbors malicious persistence. To mitigate this, design the exercise around vital services, working backwards to determine the dependencies. Identify the service critical to user authentication, the mechanism for resolving names, the data store access, and the service's reachability by users.

Document this sequence before the exercise, then test if reality aligns. Often, the order is incorrect in a predictable manner - an identity service may be restored too late, causing authentication failures; DNS records may still point to the old address, preventing clients from reaching the restored service; or a certificate required by the restored service might have been stored only on the lost system.

Identifying these discrepancies during an exercise is cost-effective, but overlooking them during a real recovery, when time is of the essence, is disastrous. Verify that the restoration process yields a clean state. A restoration replicates the backup's state at that moment. If the compromise occurred before the latest backup, the restoration reintroduces the threat.

To address this, establish how the attack's timing will be determined and how to select a restore point prior to it, without sacrificing valuable data. Immutable or write-once storage is essential to accomplish this, as a backup that an attacker can delete or encrypt is not a viable recovery option. Confirm that the immutability feature is enabled within the product, rather than relying on vendor documentation.

Exercise the decisions, not just the technology. Recovery necessitates choices that restoration scripts cannot make: choosing a restore point, deciding between rebuild and restore, and determining if a service should return before it's fully patched. These critical decisions should be exercised by the individuals who would make them, with a documented record of each choice.

Time the exercise and contrast it with the documented objective. If the target is four hours and the first attempt consumes two days, that discrepancy constitutes the finding, which is significantly more valuable than a simple pass or fail outcome. Lastly, maintain the evidence. Record the restore point and its timestamp, the components restored, the verification conducted, and any residual gaps.

Store this record alongside the backup configuration so subsequent exercises begin from the actual tested scenario, not assumptions.