Rsnapshot
I think of myself as an advanced Linux users. I have been using it exclusively for over 10 years plus and as a homelab enthusiast and a network/system administrator for my job, I use it every single day. But I hadn’t taken the time to sit down and really give rsnapshot a decent try. I have worked with rsync and scripting during all of this time but now that my needs are expanding, rsync is not keeping up with the scale. It took a little bit of time to understand how to work with the config file but once I had learned it, I implemented it on all my servers and workstations.
rsync
Before we learn about what rsnapshot is we need to understand what rsync is. rsync is as simplistic as a backup can be but with added features from the likes of the cp (copy) command. Since I mentioned cp let’s see how it compares to rsync.
$ cp -vR /home/wretchedghost /mnt/backup/cp
/home/wretchedghost -> cp/wretchedghost
/home/wretchedghost/.profile -> cp/wretchedghost/.profile
/home/wretchedghost/.bash_logout -> cp/wretchedghost/.bash_logout
...
$ rsync -av /home/wretchedghost /mnt/backup/rsync
sending incremental file list
wretchedghost/
wretchedghost/.ICEauthority
wretchedghost/.Xauthority
...
sent 2,217,369,096 bytes received 52,625 bytes 295,656,229.47 bytes/sec
total size is 2,216,574,772 speedup is 1.00
rsync took a little longer than cp to run. That is because rsync was indexing/archiving (-a) each file that was going to be placed into the /mnt/backup/rsync folder. I ran cp -vR to show verbosity and be recursive. The same is true for rsync which is the -a flag which stands for archive and run recursively and stores all permissions in one flag. -v is for verbose. Let’s run the same command but add a few new files to the home directory.
$ seq 1 1000000 > /home/wretchedghost/number_file.txt
$ seq 1 1000000 > /home/wretchedghost/number_file1.txt
I created two new files that wrote 1 to 1,000,000 each on their own lines totally 6.6MB each. Now let’s run cp and rsync again as we did above.
$ cp -vR /home/wretchedghost /mnt/backup/cp
/home/wretchedghost -> cp/wretchedghost
/home/wretchedghost/.profile -> cp/wretchedghost/.profile
/home/wretchedghost/.bash_logout -> cp/wretchedghost/.bash_logout
...
$ rsync -av /home/wretchedghost /mnt/backup/rsync
sending incremental file list
wretchedghost/
wretchedghost/number_file.txt
wretchedghost/number_file1.txt
sent 13,922,805 bytes receiving 615 bytes 27,846,840.00 bytes/sec
total size is 2,230,352,564 speedup is 160.19
Notice that cp -vR copied all the same files over again to the /mnt/backup/cp folder basically deleting each old file and replacing it with the latest copy whether it changed or not, whereas rsync only copied over the changed or new files to /mnt/backup/rsync. That is the power we get from rsync. rsync backup scheme has a few caveats though.
The backups will not scale very well as any new changes will wipe out the old files if we choose to write to the same folder every time. You can have rsync backup to a new folder each time via some scripting but you waste a ton of storage this way.
Backing up to a new folder every day or every hour means that each new folder will be a n copy of the original thus increasing the storage used on the storage device by n. (Original + today= 2 copies. Original + today + yesterday = 3 copies. Original + today + yesterday + day_before = 4 copies and so on meaning you are using n times the number of copies, which adds up very fast in terms of storage space used).
Now that we understand how rsync works lets now take a look at rsnapshot and how it works more efficiently for larger operations and scales better than rsync alone.
rsnapshot
rsnapshot is very similar to rsync since it uses rsync but with some custom scripting to do its magic. While rsync will archive and backup incrementally, rsnapshot will create snapshots, as the name implies, and store those snapshots at specific intervals that you will setup later.
After installing the rsnapshot package from your distro’s package manager you will edit and tweak the /etc/rsnapshot.conf file to control what rsnapshot can and can’t do.
:warning: Similar to how .yaml files only works with specific spacebar operations rsnapshot will not function correctly unless you use tabs exclusively and no spaces. The tabs must also be eight spaces!
Inside the /etc/rsnapshot.conf file you will need to edit a few things to get rsnapshot working.
$ vim /etc/rsnapshot.conf
#################################################
# rsnapshot.conf - rsnapshot configuration file #
#################################################
# #
# PLEASE BE AWARE OF THE FOLLOWING RULE: #
# #
# This file requires tabs between elements #
# #
#################################################
#######################
# CONFIG FILE VERSION #
#######################
config_version 1.2
###########################
# SNAPSHOT ROOT DIRECTORY #
###########################
# All snapshots will be stored under this root directory.
#
snapshot_root /var/cache/rsnapshot
# If no_create_root is enabled, rsnapshot will not automatically create the
# snapshot_root directory. This is particularly useful if you are backing
# up to removable media, such as a FireWire or USB drive.
#
#no_create_root 1
...
(output is shortened)
You will notice the warning about tabs right at the beginning. Remember this! Now we need to start by editing a few things.
snapshot_root /var/cache/rsnapshot tells rsnapshot where to place the files that are being backed up. Most likely you will want to change this to something more meaningful. In my case I changed it to /backup that I created and setup proper permission for my user.
$ sudo mkdir /backup
$ sudo chown -R wretchedghost:wretchedghost /backup
Here is my modified /etc/rsnapshot.conf file.
config_version 1.2
snapshot_root /rsnapbackup/
cmd_cp /usr/bin/cp
cmd_rm /usr/bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
interval hourly 12
interval daily 7
interval weekly 4
interval monthly 6
verbose 2
loglevel 3
logfile /var/log/rsnapshot
lockfile /var/run/rsnapshot.pid
exclude /home/wretchedghost/.cache
exclude /home/wretchedghost/.steam
exclude /home/wretchedghost/.steampath
exclude /home/wretchedghost/.steampid
exclude /home/wretchedghost/.local/share/Steam
exclude /home/wretchedghost/.ignition
exclude /home/wretchedghost/.wine
exclude /home/wretchedghost/.java
exclude /home/wretchedghost/.factorio
exclude /home/wretchedghost/.mozilla
exclude /home/wretchedghost/.thunderbird
exclude /home/wretchedghost/Downloads/*.iso
exclude /etc/passwd
exclude /etc/ufw
exclude /etc/shadow*
exclude /etc/openvpn
exclude /etc/conf.d/*
exclude /etc/ipsec*
exclude /etc/teamviewer
exclude /etc/sudoers*
exclude /etc/polkit-1
exclude /etc/audit
exclude /etc/cups
exclude /etc/NetworkManager
exclude /etc/ca-certificates
exclude /etc/default
exclude /etc/pacman.d*
exclude /etc/systemd*
exclude /etc/alsa
exclude /etc/ssl*
exclude /etc/fonts
exclude /etc/gshadow*
exclude /etc/crypttab
exclude /etc/.pwd.lock*
exclude /etc/libaudit.conf
exclude /etc/xdg*
exclude /etc/mtab
exclude /etc/localtime
exclude /etc/os-release
exclude /etc/iptables*
exclude /etc/ppp*
exclude /etc/ssh*
exclude /etc/webapps*
###############################
### BACKUP POINTS / SCRIPTS ###
###############################
backup /home/ sys76/
backup /etc/ sys76/
I have lots of excludes since steam and several other programs like to store their files not only in their own folders but also in my home directory as well, which I think is dumb but what can you do.
You will also notice that I changed the alpha, beta, etc, to something more meaningful to me. I prefer to use the simple words of hourly, daily, weekly, monthly.
Lastly you will want to edit the backup points/scripts to the place you want your backups to be stored.
The forward slashes on the second column (ie, /home/ and /etc/) are very important. The third column represents the name of the master folder that will be created and where it will be saving everything in. In this case it will be under the /backup/sys76/ folder since sys76/ will be using the snapshot_root that we setup in line two of the config. Notice the no trailing forward slash on sys76/ but there is a slash at the end.
Let’s setup automation now via cron.
Automating rsnapshot
Instead of building a script and placing it in crontab I prefer to use the /etc/cron.x sub-folders due to their transportability. I have found that with crontab all the commands and scripts placed in it are sent to /var/spool/cron/crontabs which only root has read and write access to. That is OK when I’m able to pull the config and back it up for later but since I disable all root SSH access on all of my boxes, sending that file to other servers is rather tedious to setup and I find it easier to place scripts that only need to run on a scheduled times like hourly, daily, etc., in the /etc/cron.x folders.
If you do need to run a script twice a day or every few minutes then its easier to do with crontab alone and not use the /etc/cron.x locations.
First we will start with the hourly rsnapshot interval. Lets make sure our /etc/rsnapshot works correctly by running rsnapshot configtest.
$ rsnapshot configtest
Syntax OK
Now let’s create the script for hourly. You can call it whatever you want and you don’t have to put a dot extension since Linux knows what to do with it.
$ sudo vim /etc/cron.hourly/rsnapshot_sys76
#!/bin/bash
/usr/bin/rsnapshot hourly > /dev/null 2>&1
We placed a > /dev/null 2>&1 redirect to /dev/null and redirect/merge stderr to stdout, since the script does not run correctly without it. All you have to do is the same thing for /etc/cron.daily, /etc/cron.weekly, and /etc/cron.monthly changing only the hourly tag to their corresponding backup schedule.
You can manually run a rsnapshot backup but you must have all of the lowest snapshots done before another can be run, (ie., if you have daily set to 7 then you must fill up daily.0 - daily.6 before you can run rsnapshot weekly).
$ /usr/bin/rsnapshot hourly
Again nothing will verify that anything has changed but if we make our way to the /backup/sys76 folder you will notice that /home and /etc directories have been backed up.
$ ls /backup/
hourly.0
Run hourly again and you will see that it has moved hourly.0 to hourly.1 and has created a new hourly.0. All of the previous files will still be in the newest hourly.0 but only those that have changed will be needed to be backed up.
$ /usr/bin/rsnapshot hourly
$ ls /backup
hourly.0 hourly.1
Conclusion
Using rsync is a great tool but working with rsnapshot allows for backups and versioning control for as long of a duration as you have them set to. Virtually you could keep all backups til the end of time if you tell rsnapshot to keep so many x versions. It offers great scaling capabilities and overall is not too difficult once you learn its little intricacies.