Backing Up Data
Finn Greig
Sorting out redundancy for physical stuff is easy - you either just get contents insurance or you could buy multiple units of an item and keep the rest of them in self-storage or any other separate location to your usual residence. Data is much different - you can't just buy an insurance policy for the data on your computer, call it day and sleep soundly at night - it requires a bit of creative planning, some legwork, automating as much as you can and diligently following a schedule for the rest.
The 3-2-1 Backup Rule
A 3-2-1 strategy means having at least three total copies of your data, two of which are local but on different mediums (read: devices), and at least one copy off-site. Well use kitten.jpg as an example for this scenario. Kitten.jpg lives on your computer at home; it was a picture that you took of your cat in 2012. Thats one copy of the data. You also have an external hard drive that you use for backing up your computer; if youre on a Mac, you might be using it as a Time Machine drive (and Backblaze loves Time Machine). As part of its backup process, that external hard drive will back up kitten.jpg. Thats a second copy, on a different device or medium. In addition to that external hard drive, you also have an online backup solution. The online backup continuously scans your computer and uploads your data off-site to a data center. Kitten.jpg is included in this upload, and that becomes the third copy of your data. ~ Backblaze
3-2-1 was considered to be the golden standard of backup strategies for a long time, however, since then a lot has changed - new standards of physical media and services (e.g. cloud object stores) have come about, and improved redundancy can be achieved with minimal additional overheads. In this guide, I will be implementing a strategy similar to this one suggested by Unitrends.
Getting It Done
1. What You'll Need
There are a few things you will need to implement this strategy:
Hardware
- 2x external hard drives. I picked up a couple of these for myself: https://www.pbtech.co.nz/product/HDDSEA0423/Seagate-2TB-Expansion-Portable-Hard-Drive. You can choose any capacity you want, just make sure that there will be plenty of room for the future, and if you wanted to you could get something nicer (perhaps ruggedised with USB-C, like a LaCie).
Software
- Restic - https://restic.net/. This will backup the data to your various media - it's cross-platform, handles encryption, uses diffs for efficient backups, has verification to ensure file integrity and most importantly it's open source.
Services
Backblaze B2 - https://www.backblaze.com/b2/cloud-storage.html. This is the cloud service that will be used for intercontinental off-site backups.
Create two accounts, one for the US region and one for the EU region. You can use the same email address by appending +b2eu to the username of your email address when creating the EU account.
A safety deposit box. There will be different businesses in different locations offering these, however, at least in Wellington there is one offering - NZ Vault (http://www.nzvault.co.nz/safety-deposit-boxes/wellington-vault-options/). Box type B would be the cheapest option, as type A cannot fit many external hard drives as it is too narrow. The boxes are 50cm in length which is ample for our purposes - you could utilise the spare space however you wish. I would recommend using it to store copies of important identity and insurance documents, backup U2F security keys and anything else precious (e.g. bullion).
2. Setting It Up
Format the external hard drives as exFAT - this will allow the drive to be most compatible with various operating systems.
Create a new Restic repository on each of them:
restic init --repo /path_to_drive/restic
- Ensure you have setup CLI credentials for B2 in the US Region:
export B2_ACCOUNT_ID=<MY_APPLICATION_KEY_ID> export B2_ACCOUNT_KEY=<MY_APPLICATION_KEY>
- Create a Restic repository in the US B2 bucket:
restic -r b2:us_bucket_name:path/to/repo init
- Setup the CLI credentials again but for B2 in the EU Region:
export B2_ACCOUNT_ID=<MY_APPLICATION_KEY_ID> export B2_ACCOUNT_KEY=<MY_APPLICATION_KEY>
- Create a Restic repository in the EU B2 bucket:
restic -r b2:eu_bucket_name:path/to/repo init
3. Backing Up
Create a directory somewhere for your scripts and exclude file.
Inside that directory, create an excludes.txt. This is like a .gitignore file and specifies files or directories to exclude from your backup. Here is an example:
ntuser.dat *.rs node_modules C:\Users\UserName\AppData
- Create scripts to make backing up convenient. For mine I use PowerShell but it should be easy enough to write a shell script if you're using Linux/BSD/macOS. Here are some examples:
On-site HDD:
`$password = Read-Host "Enter password" -AsSecureString $env:RESTIC_PASSWORD = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)) restic -r F:\backup backup --use-fs-snapshot C:\A\Folder D:\Another\Folder --exclude-file=excludes.txt`Off-site HDD:
`$password = Read-Host "Enter password" -AsSecureString $env:RESTIC_PASSWORD = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)) restic -r G:\backup backup --use-fs-snapshot C:\A\Folder D:\Another\Folder --exclude-file=excludes.txt`B2 US:
`$env:B2_ACCOUNT_ID = "YOUR_US_ACCOUNT_ID" $key = Read-Host "Enter key" -AsSecureString $env:B2_ACCOUNT_KEY = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($key)) $password = Read-Host "Enter password" -AsSecureString $env:RESTIC_PASSWORD = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)) restic -r b2:us_bucket_name:backup backup --use-fs-snapshot C:\A\Folder D:\Another\Folder --exclude-file=excludes.txt`B2 EU:
`$env:B2_ACCOUNT_ID = "YOUR_EU_ACCOUNT_ID" $key = Read-Host "Enter key" -AsSecureString $env:B2_ACCOUNT_KEY = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($key)) $password = Read-Host "Enter password" -AsSecureString $env:RESTIC_PASSWORD = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)) restic -r b2:eu_bucket_name:backup backup --use-fs-snapshot C:\A\Folder D:\Another\Folder --exclude-file=excludes.txt`- Run these scripts every now and then, and when your off-site backup has finished at least once you can place it into your safety deposit box, and retrieve it later when you want to do another backup then put it back.