Test the restore, not the backup

Ask a team whether their data is backed up and the answer is almost always yes. Snapshots run nightly, retention is set, the console shows a row of green checks. Ask when they last restored one into a working environment and the room gets quieter.

Those are different questions. A backup job reporting success tells you a file was written somewhere. It does not tell you the file is complete, that anyone still has the key to read it, or that the application will start when it is pointed at the result. The only way to know those things is to do the restore, on purpose, before the day you need it.

What a green backup does not prove

We have watched teams discover each of these during an incident, which is the worst time to discover anything:

• The database was covered, but the uploaded files lived in a bucket nobody added to the policy. The records came back pointing at documents that no longer existed.
• The snapshot was encrypted with a key whose permissions changed months ago. The backup was intact and unreadable.
• The data restored cleanly, but the application expected a schema from a later migration, a secret that was never stored with the infrastructure, or a queue that had to be drained first.
• The restore worked and took most of a day. The recovery objective everyone had agreed to was an hour.

None of these show up in a backup report, because the report is answering a question about storage. The question that matters is about the application.

Define “restored” before you drill

A restore drill needs a finish line, or it ends when the data copy finishes and everyone declares victory. We write the finish line down in terms a user would recognize: the application is running against restored data, someone can sign in, and a handful of named checks pass — a recent record is present, an attached file opens, a report totals to what it did before.

Then time it. The clock starts when someone decides to restore and stops when those checks pass, not when the snapshot finishes copying. That number is your real recovery time, and it is usually the first honest one the team has seen. If it is longer than the number in the continuity plan, the plan is the thing that is wrong, and now you can fix it with evidence.

Make it a scheduled drill, not a heroic act

The restore should run into a separate, disposable environment, on a calendar, with a named owner. Quarterly is a reasonable start for most systems; anything that changes shape often deserves more. The output is short: how long it took, which checks passed, and what broke. Anything that broke becomes a ticket with the same priority as a production bug, because it is one that has not happened yet.

This is far easier when the environment itself comes from code. If a fresh copy of the system can only be built by hand, every drill is also an archaeology project. We wrote about that half of the problem in one path to production; a restore drill is where you find out whether it held.

Two habits keep the drill honest. Rotate who runs it, so the procedure is written down well enough for someone who did not write it. And restore from the real backups, through the real permissions, rather than a copy someone prepared the day before.

What to ask this week

Three questions will tell you where you stand. When did we last restore production data into a working environment? How long did it take, from decision to a passing check? Who owns doing it again, and when? If any answer is a shrug, the backups are a hope rather than a plan.

We build restore drills into our cloud, DevOps and platform work alongside the pipeline and the monitoring, because a system is only as recoverable as the last time someone actually recovered it.