Restore or Regret: Seven Recovery Rehearsals That Prove Your Backups Are Real

Backup dashboards report green because the software completed a write. They say nothing about whether the data is readable, whether the credentials to restore it still work, or whether the person who knows the sequence is on a plane. Recovery capability is a claim that has to be demonstrated on a schedule, in front of witnesses, with evidence retained.

The seven drills below are ordered roughly from least to most disruptive. Each includes the scenario, the failure mode it exposes, and the artifact an IT team should capture for auditors, regulators, and the board.

Drill 1: Bare-Metal Restore to Dissimilar Hardware

Scenario. Take a Tier-1 application server with no current rebuild documentation and restore it to a different model, a different hypervisor host, or a cloud instance, using only what exists in the backup catalog.

Failure mode exposed. Missing drivers, absent storage controller firmware, incorrect boot configuration, and the uncomfortable discovery that the backup contains data volumes but not the system state required to boot. This drill also surfaces rebuild documentation that has drifted since the last hardware refresh.

Evidence to capture. Restore start and completion timestamps, the hardware or instance type used, screenshots of the application health check, and a written list of every manual step the engineer had to improvise.

Drill 2: SaaS Data Recovery Inside the Retention Window

Scenario. Restore a deleted mailbox, a SharePoint document library with version history intact, and a Teams channel conversation from a third-party backup tool, then compare the result with what native retention would have provided on its own.

Failure mode exposed. Native retention windows in Microsoft 365 and Google Workspace are frequently shorter than legal hold obligations, and licensing tiers silently change what is recoverable. Many organizations discover that their SaaS backup covers mail but not channel messages, or that the backup service account lost its permissions during a tenant consolidation.

Evidence to capture. The restore point used, the retention policy in force at the time of deletion, the license tier assumed, and the elapsed time to a fully restored object including indexing.

Drill 3: Identity Infrastructure Recovery

Scenario. In an isolated environment, restore a domain controller and authoritative directory data, or recover deleted objects and configuration in a cloud identity tenant. Include a test of the break-glass administrative account and its documented retrieval procedure.

Failure mode exposed. Directory recovery has ordering constraints that general-purpose backup tools do not enforce. Restoring a domain controller from an image taken outside the supported tombstone lifetime can reintroduce deleted objects and corrupt replication. Cloud tenants introduce a second failure mode: recovery depends on an administrative account whose credentials were stored in the system being recovered.

Evidence to capture. The recovery method used, tombstone and recycle bin settings at time of test, the timestamp of the last successful privileged account validation, and confirmation that break-glass credentials remain reachable through an independent channel.

Drill 4: Network Configuration Rebuild

Scenario. Rebuild a core switch stack or a perimeter firewall from configuration backups in a lab or maintenance window, then verify that routing adjacencies, access control lists, VPN tunnels, and licensing all come back correctly.

Failure mode exposed. Configuration archives that omit certificates, license files, or out-of-band access settings. The most common finding is that the backup lives on a server reachable only through the device being rebuilt.

Evidence to capture. A configuration hash comparison between pre-failure and post-restore, the firmware version applied, and the measured time to full routing convergence.

Drill 5: Ransomware Recovery From an Isolated Copy

Scenario. Simulate encryption of a production file server and its backup catalog, then recover from an immutable or air-gapped copy into a clean-room network segment. Scan the restored data before it rejoins production.

Failure mode exposed. Backup infrastructure that shares credentials, domain membership, or management networks with production. If the backup console can be reached with a stolen domain administrator credential, the immutable copy may be intact while the catalog, job schedules, and restore credentials are gone.

Evidence to capture. The isolation architecture in place at time of test, the age of the recovery point objective actually achieved, malware scan results on restored data, and the total elapsed recovery time objective measured from detection to verified application availability.

Drill 6: Point-in-Time Database Consistency

Scenario. Restore a production database to an arbitrary point within the transaction log window, then run application-level integrity checks rather than database-level checks alone.

Failure mode exposed. Backups that are individually valid but mutually inconsistent, particularly with high-frequency snapshots across multiple volumes. Application referential integrity, not just database page checksums, is where recoverability breaks.

Evidence to capture. The requested point in time, the achieved point in time, the integrity check output, and the reconciliation performed against an external system of record.

Drill 7: The People Drill

Scenario. Run the restore using only the on-call rotation and the written runbook, while the primary backup administrator is unavailable by design.

Failure mode exposed. Institutional knowledge concentrated in one or two people, runbooks that assume undocumented context, and escalation paths that terminate at someone who left the company two reorganizations ago.

Evidence to capture. The runbook revision used, the number of steps that required improvisation, the time to first successful restore action, and a prioritized list of documentation defects.

Turning Drills Into Assurance

Schedule the drills quarterly, rotate the personnel, and keep a single register that records scenario, result, elapsed time, defects found, and remediation owner. Executives do not need a backup success rate; they need the date of the last verified restore and the measured time to recover. Auditors will ask for both, and the register is the fastest way to answer.