What do you all do to backup all your data? Any recommendations for a better solution?
The directory structure is:
+-- Machine1 -- +-- date0a --+-- Data
| +-- date1a --+-- Data
| +-- date2a --+-- Data
| +-- ...
|
+-- Machine2 -- +-- date0b --+-- Data
+-- date1b --+-- Data
+-- date2b --+-- Data
+-- ...
So for each machine there is a directory containing the backups. The backups are named as the date/time when they were taken. This would seem slow and wasteful, but each backup has hardlinks - where possible -- to the previous backup. So the space taken by each new backup is only the files that have changed, but each backup can be copied as a whole, and doesn't need to be reconstructed in any way.It has the benefits of differential backups, but without having to restore a complete checkpoint and then roll forward with the updates.
rsync has options to do most of this for you. I can provide scripts if you're interested, but I'm sure others will comment here will more slick and tidy solutions. I do this because I understand it completely, it meets my needs, and the backups are completely under my control.
I had a disk failure in February, regular backups saved me. I'm sorry you've had this experience, and wish you luck in recovering.
Three tricks that have worked for me:
1) Reduce the resistance in your brain to making backups
Easy wins can be made here without changing anything. Keep the external drive ready to go on your desk at any time, have it start backing up as soon as you plug it in, etc. Reduce the dissonance you feel in your brain to making backups regularly.
2) Automate it as much as possible
Take the resistance entirely out of the equation by automating it. For example, in your case you might want to try plugging the external drive into an always-on raspberry PI, and use Syncthing to constantly send your files there.
3) Have a process that keeps you accountable
You need something to make you feel guilty for not having backups, that overpowers the resistance in your brain with shame! Even a recurring calendar entry can work well.
I've taken it a step further by writing software to solve this very problem. I've seen and heard about way too many backup failures in my short career, and determined to help bring good backup practices to more people. Check out the idea at https://backupshq.com - we'll be launching privately soon and always interested in feedback on the idea.
This works for me because it balances effort with the degree to which I care about my data. It's not much work because I don't care that much. The reason I don't care that much is because data is a burden. Data implies potential work of the odious chore variety.
Losing data sucks for a little while. Managing archives sucks forever.
* Done regularly in an automated fashion (if it’s manual, you will forget); use what comes with your OS or buy a good paid backup application
* Keeps older versions with a particular scheme (if you only sync, then your backups will pick up bit rot, and unintended deletions which are manual or through malware)
* Have multiple backups on different media and different locations (external drives break, cloud services lose data, your friend’s house may get burgled); you must have local copies (fast) and online copies (physically remote)
I have backups on external drives, a NAS and a cloud backup.
Now my laptop syncs my Documents directory to google drive and Synology picks it up from there. At any given point in time, I have 3 copies of my most critical files.
And the best part is after the setup, I don't have to think about it.