« Copy between files in vi/vim «
» Clone or Change your MAC address in Linux »

HOWTO: Backup nightly via rsync

rsync is a software application for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression and recursion.

In daemon mode, rsync listens to the default TCP port of 873, serving files in the native rsync protocol. rsync can also be used to synchronize local directories, or via a remote shell such as RSH or SSH. In the latter case, the rsync client executable must be installed on both the local and the remote host.

Purpose…
To backup your entire ‘home’ directory (or any directory of your choice), nightly (or at a time of your choosing), efficiently (low bandwidth) and securely (encrypted). It will also produce compressed logs in a directory of your choice. If you follow this tutorial and set this up correctly, you should be able to “set it and forget it”.

If you’re looking to synchronize two computers (office and home computer maybe) then try Unison. ($ sudo aptitude install unison unison-gtk)

What programs will be used?
rsync and OpenSSH.

Behind the commands & scripts…
These commands will (in order):

  • generate a public/private key on the client machine in /home/$USER/.ssh/backup
  • send the public key to the backup server
  • create a backup folder
  • configure the SSH key on the server
  • test if the public/private key pair are working (is it asking for a password?)
  • create a script to run rsync via ssh and port 2222
  • create a crontab that will run the script nightly (or at a time of your choosing)
  • create a log file and compress it

Assumptions

  • Your backup server’s IP is 192.168.1.2
    • this can be any IP address whether internal (192.168.1.*), external (24.235.53.*) or a DNS (google.ca)
  • Both your server and you client computer have SSH installed ($ sudo aptitude install ssh)
  • You’ve changed the SSH port to 2222 (/etc/ssh/sshd_config)
    • if you want to keep SSH at its default port of 22, simply remove ‘-p 2222′ or ‘-P 2222′ from all of my commands
  • You’ve created an identical user on your server to the one you’re backing up (same username — the password is allowed to be different)
    • example: if my username is ‘brett’ on the client-side, make a user called ‘brett’ on your backup server

Setting up the client-side
Step 1: Generate a public/private key on the client machine in /home/$USER/.ssh/backup

Code:
ssh-keygen -t dsa -f /home/$USER/.ssh/backup
Quote:
Generating public/private dsa key pair.
Created directory ‘/home/$USER/.ssh’.
Enter passphrase (empty for no passphrase): <empty>
Enter same passphrase again: <empty>
Your identification has been saved in /home/$USER/.ssh/backup.
Your public key has been saved in /home/$USER/.ssh/backup.pub.
The key fingerprint is:
$FINGERPRINT $USER@$HOSTNAME

Step 2: Secure copy the public key to your backup server

Code:
scp -P 2222 /home/$USER/.ssh/backup.pub $USER@192.168.1.2:/home/$USER/
Quote:
$USER@192.168.1.2’s password: <$USER’s password>
backup.pub 100% 605 0.6KB/s 00:00

Step 3: Create the folder ‘backup’ if it doesn’t exist, the folder ‘.ssh’ if it doesn’t exist and configure the public key to OpenSSH’s liking

Code:
ssh -p 2222 $USER@192.168.1.2 "if [ ! -d backup ]; then mkdir backup; fi ; if [ ! -d .ssh ]; then mkdir .ssh ; chmod 700 .ssh ; fi ; mv backup.pub .ssh/ ; cd .ssh/ ; if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi ; cat backup.pub >> authorized_keys ; rm backup.pub;"
Quote:
$USER@192.168.1.2’s password: <$USER’s password>

Step 4: Test if you can connect without being prompted for a password

Code:
ssh -p 2222 -i /home/$USER/.ssh/backup $USER@192.168.1.2
Quote:
$USER@$SERVER_HOSTNAME: $

Note:
You should now be connected to your server without being prompted for a password.

Step 5: Create & save the ‘rsync.sh’ script

Instructions:
Save this as ‘rsync.sh’ wherever you keep your cron scripts. If you don’t have such a directory, save it where it will never be moved or deleted.

Code:
#!/bin/sh

# %Y     year
# %m     month (01..12)
# %d     day of month (e.g, 01)
# %s     seconds since 1970-01-01 00:00:00 UTC

# edit these variables to your specific needs
USER=$USER # unfortunately, cron does know the variable $USER, so this has to be edited by hand
BACKUPDIR=/home/$USER/
THEDATE=`date '+%Y%m%d-%s'` # 20071010-1192044000
IPADDRESS=192.168.1.2
LOGDIR=/home/$USER/my_backups/logs
EXCLUDEFILE=/home/$USER/cron/exclude.txt
LOGFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.log
GZFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.gz
SSHPORT=2222

if [ ! -d $LOGDIR ]; then
mkdir -p $LOGDIR
fi

# -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
# -v, --verbose               increase verbosity
# -r, --recursive             recurse into directories
# -z, --compress              compress file data during the transfer
#     --delete                delete extraneous files from dest dirs
#     --log-file=FILE         log what we're doing to the specified FILE
# -e, --rsh=COMMAND           specify the remote shell to use

rsync -avrz --delete --delete-excluded --exclude-from=$EXCLUDEFILE --log-file=$LOGFILE --rsh="ssh -p $SSHPORT -i /home/$USER/.ssh/backup" $BACKUPDIR $USER@$IPADDRESS:/home/$USER/backup/

# gunzip the logfile
gzip -c $LOGFILE > $GZFILE

# delete the original, uncompressed logfile
rm $LOGFILE

Step 6: Edit the ‘rsync.sh’ variables

Instructions:
Edit these variables in ‘rsync.sh’ to your liking, or keep the default. You MUST however, edit ‘USER=$USER’

Code:
USER=$USER # unfortunately, cron does know the variable $USER, so this has to be edited by hand
BACKUPDIR=/home/$USER/
THEDATE=`date '+%Y%m%d-%s'` # 20071010-1192044000
IPADDRESS=192.168.1.2
LOGDIR=/home/$USER/my_backups/logs
EXCLUDEFILE=/home/$USER/cron/exclude.txt
LOGFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.log
GZFILE=/home/$USER/my_backups/logs/rsync-$THEDATE.gz
SSHPORT=2222

Optional: Create an exclude list

Instructions:
Create a simple text file that lists every directory you want rsync to skip/exclude, putting each directory on a new line. This affects this variable:

Code:
EXCLUDEFILE=/home/$USER/cron/exclude.txt

Step 7: Install a crontab

Code:
export EDITOR=vim && crontab -e
Quote:
* 3 * * * /home/<username>/<wherever you put ‘rsync.sh’>/rsync.sh

Notes:

  • Unfortunately, you can not use the variable $USER here
  • If you rather use a graphical editor, replace ‘vim’ with ‘gedit’, ‘kate’ or ‘mousepad’ (depending on your distribution)

If all goes as it should, this should only have taken 10 minutes to setup.

Related Posts

Tags: ,

This entry was posted on Thursday, May 29th, 2008 at 1:49 pm and is filed under Networking. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a Reply