Also, if applicable, have a different person perform the restore every time and have them do it just by following the documentation. This way multiple persons have actual experience with the process if the shit ever hits the fan and this also makes sure the documentation is accurate and up-to-date.
I started out in a big iron shop in the 90s. I was in charge of backups. We had an outage after a power loss, and the generator not kicking in. Our local backup set didn’t restore. The basement set, in a fire/waterproof safe didn’t restore. The off-site set restored.
The hours between the first fail and success were pure terror.
3-2-1...
Three copies of your data, in two physical locations, equals one backup. And it's not real until you do a test restore yes indeed!
My understanding of 3-2-1 is different;
3 copies of your data, on two different storage mediums, with one offsite.
E.g. SSD live copy, hot HDD backup, cold HDD backup offsite.
Is this wrong?
Afaik that's the common read on 3-2-1 though im wondering these days if the "separate mediums" still is that relevant to me it means storing data on different types of disks however in the end all is zfs for me so not really a different medium (since same file system)? Anyway I still have to set up my off-site backups anyway to adhere to 3-2-1 :D
We do a restore test once a quarter, but we are SOC2 so we must do it. Doesn't have to restore an entire VM, just a random used file.
A few months after setting up the backups for my server to two remote locations and patting myself in the back for it, I woke up in the middle of the night realizing I had no idea how to restore.
I worked at a datacenter that wanted to change backup vendors, as we dug into the details we found out that the agent based backups needed an agent running on the machine to restore to and they didn't have a linux agent. Despite this obvious problem mgmt chose this vendor. It didn't take long before sysadmins were rebuilding linux boxes from scratch in the wee hours. I left shortly after.
Ain't got time for that
Instead, I let my faith carry me though
Rules of Tech Support
Rule T9C - If you can't restore from it, you don't have a backup.
Rule T9D - If you haven't tested your backup recently, you don't have a backup.
How do you test one
It depends on the type of backup:
For a filesystem backup, restore one or more files to a secondary location. E.g. pick a few files out of the backup and try to restore them to a temporary folder. Then hash the original and restored files to verify integrity.
For a full machine backup (e.g. VM backup), restore a copy of the machine to a test location. Spin up the test machine to verify that it can boot.
For a database backup, restore a copy of the database to a test location (e.g. change the database name as part of the restore process), compare a few tables against the real database to verify integrity.
Pretty much, it's going to be some version of "Restore X to a test location and verify integrity". You want to both prove that the backup can be restored and that the restored copy is actually intact.
Thanks for explaining this! Very helpful and easy to understand. Do you have preferred programs for the two actions? I currently just rsync my servers.
I use restic to back up files to Wasabi.
It's all scripted but the steps are:
A community dedicated to the profession of IT Systems Administration
No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world