HOWTO: Backup nightly via rsync
rsync is a software application for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression and recursion.
In daemon mode, rsync listens to the default TCP port of 873, serving files in the native rsync protocol. rsync can also be used to synchronize local directories, or via a remote shell such as RSH or SSH. In the latter case, the rsync client executable must be installed on both the local and the remote host.
Purpose…
To backup your entire ‘home’ directory (or any directory of your choice), nightly (or at a time of your choosing), efficiently (low bandwidth) and securely (encrypted). It will also produce compressed logs in a directory of your choice. If you follow this tutorial and set this up correctly, you should be able to “set it and forget it”.
If you’re looking to synchronize two computers (office and home computer maybe) then try Unison. ($ sudo aptitude install unison unison-gtk)
What programs will be used?
rsync and OpenSSH.
Behind the commands & scripts…
These commands will (in order):
- generate a public/private key on the client machine in /home/$USER/.ssh/backup
- send the public key to the backup server
- create a backup folder
- configure the SSH key on the server
- test if the public/private key pair are working (is it asking for a password?)
- create a script to run rsync via ssh and port 2222
- create a crontab that will run the script nightly (or at a time of your choosing)
- create a log file and compress it
Assumptions
- Your backup server’s IP is 192.168.1.2
- this can be any IP address whether internal (192.168.1.*), external (24.235.53.*) or a DNS (google.ca)
- Both your server and you client computer have SSH installed ($ sudo aptitude install ssh)
- You’ve changed the SSH port to 2222 (/etc/ssh/sshd_config)
- if you want to keep SSH at its default port of 22, simply remove ‘-p 2222′ or ‘-P 2222′ from all of my commands
- You’ve created an identical user on your server to the one you’re backing up (same username — the password is allowed to be different)
- example: if my username is ‘brett’ on the client-side, make a user called ‘brett’ on your backup server
Setting up the client-side
Step 1: Generate a public/private key on the client machine in /home/$USER/.ssh/backup
ssh-keygen -t dsa -f /home/$USER/.ssh/backup
| Generating public/private dsa key pair. Created directory ‘/home/$USER/.ssh’. Enter passphrase (empty for no passphrase): <empty> Enter same passphrase again: <empty> Your identification has been saved in /home/$USER/.ssh/backup. Your public key has been saved in /home/$USER/.ssh/backup.pub. The key fingerprint is: $FINGERPRINT $USER@$HOSTNAME |
Step 2: Secure copy the public key to your backup server
scp -P 2222 /home/$USER/.ssh/backup.pub $USER@192.168.1.2:/home/$USER/
| $USER@192.168.1.2’s password: <$USER’s password> backup.pub 100% 605 0.6KB/s 00:00 |
Step 3: Create the folder ‘backup’ if it doesn’t exist, the folder ‘.ssh’ if it doesn’t exist and configure the public key to OpenSSH’s liking
ssh -p 2222 $USER@192.168.1.2 "if [ ! -d backup ]; then mkdir backup; fi ; if [ ! -d .ssh ]; then mkdir .ssh ; chmod 700 .ssh ; fi ; mv backup.pub .ssh/ ; cd .ssh/ ; if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi ; cat backup.pub >> authorized_keys ; rm backup.pub;"
| $USER@192.168.1.2’s password: <$USER’s password> |
Step 4: Test if you can connect without being prompted for a password
ssh -p 2222 -i /home/$USER/.ssh/backup $USER@192.168.1.2
| $USER@$SERVER_HOSTNAME: $ |
Note:
You should now be connected to your server without being prompted for a password.
Step 5: Create & save the ‘rsync.sh’ script
Instructions:
Save this as ‘rsync.sh’ wherever you keep your cron scripts. If you don’t have such a directory, save it where it will never be moved or deleted.
#!/bin/sh # %Y year # %m month (01..12) # %d day of month (e.g, 01) # %s seconds since 1970-01-01 00:00:00 UTC # edit these variables to your specific needs USER=$USER # unfortunately, cron does know the variable $USER, so this has to be edited by hand BACKUPDIR=/home/$USER/ THEDATE=`date '+%Y%m%d-%s'` # 20071010-1192044000 IPADDRESS=192.168.1.2 LOGDIR=/home/$USER/my_backups/logs EXCLUDEFILE=/home/$USER/cron/exclude.txt LOGFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.log GZFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.gz SSHPORT=2222 if [ ! -d $LOGDIR ]; then mkdir -p $LOGDIR fi # -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) # -v, --verbose increase verbosity # -r, --recursive recurse into directories # -z, --compress compress file data during the transfer # --delete delete extraneous files from dest dirs # --log-file=FILE log what we're doing to the specified FILE # -e, --rsh=COMMAND specify the remote shell to use rsync -avrz --delete --delete-excluded --exclude-from=$EXCLUDEFILE --log-file=$LOGFILE --rsh="ssh -p $SSHPORT -i /home/$USER/.ssh/backup" $BACKUPDIR $USER@$IPADDRESS:/home/$USER/backup/ # gunzip the logfile gzip -c $LOGFILE > $GZFILE # delete the original, uncompressed logfile rm $LOGFILE
Step 6: Edit the ‘rsync.sh’ variables
Instructions:
Edit these variables in ‘rsync.sh’ to your liking, or keep the default. You MUST however, edit ‘USER=$USER’
USER=$USER # unfortunately, cron does know the variable $USER, so this has to be edited by hand BACKUPDIR=/home/$USER/ THEDATE=`date '+%Y%m%d-%s'` # 20071010-1192044000 IPADDRESS=192.168.1.2 LOGDIR=/home/$USER/my_backups/logs EXCLUDEFILE=/home/$USER/cron/exclude.txt LOGFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.log GZFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.gz SSHPORT=2222
Optional: Create an exclude list
Instructions:
Create a simple text file that lists every directory you want rsync to skip/exclude, putting each directory on a new line. This affects this variable:
EXCLUDEFILE=/home/$USER/cron/exclude.txt
Step 7: Install a crontab
export EDITOR=vim && crontab -e
| * 3 * * * /home/<username>/<wherever you put ‘rsync.sh’>/rsync.sh |
Notes:
- Unfortunately, you can not use the variable $USER here
- If you rather use a graphical editor, replace ‘vim’ with ‘gedit’, ‘kate’ or ‘mousepad’ (depending on your distribution)
If all goes as it should, this should only have taken 10 minutes to setup.