Backup with rsync using SSH Tunneling

For those of you that read my blog often, you know that I admin the cluster that our research group uses here at CU Boulder.  Because of this, I get a lot of questions from users who don’t want to take the time to solve their own problems.  Fairly recently, our RAID-6 crashed (we had a 4th drive die and had to rebuild the array).  Normally this wouldn’t be very much of a problem as most of the files saved on our storage drive are just input files that we can re-download from a separate server, or so I thought.  Personally, all my source code is in my home folder, backed up on our data server, and backed up onto my personal laptop.  For researchers in our group who are developing code, not having a backup of source code can lead to many many months of lost work.  Well, as it turns out, many of the people in our group had their source code on our data server (the one that crashed), without a backup anywhere.  So months of work had been lost.  Well, after the rebuild I have gotten many questions on how to set up an ssh tunnel so that they can backup from our cluster, through the front end, to their home computer.

Anyway, here is how I was able to setup a passthrough on my system.  It should be noted that I have enabled password-less SSH to the frontend and from the frontend to the server.

ssh -fN -L 2222::22 @

# Transfer from cluster to local
scp -o"Port=2222" @localhost:/path/on/cluster/ /path/on/personal/computer/
# Transfer from local to prospero
scp -o"Port=2222" /path/on/personal/computer/ @localhost:/path/on/cluster/

In the above ssh command, the flags are as follows:

• -f: Requests ssh to go to background just before command execution
• -N: Do not execute a remote command
• -L [port:host:host port]: Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side

Example: (This is a purely artificial scenario, as there is no server cluster.mattdturner.com and there would be no need to go through google.com) If you want to access a cluster (cluster.mattdturner.com) but you can only access the server by first accessing a frontend (google.com), you would use the following command.

ssh -fN -L 2222:cluster.mattdturner.com:22 username@google.com

Then, to download a file (text.file) in your home directory on cluster.mattdturner.com to your local home directory,

scp -o"Port=2222" username@localhost:~/text.file ~/

Now that you are familiar with SSH Tunneling, lets setup an rsync to utilize the SSH tunnel to backup certain files from a remote server. In my case, I setup a c-shell script that is run each night at midnight to backup all of my source code. The script is below:

#!/bin/csh

date

rsync --delete -avz -e "ssh mturner@ ssh" mturner@:/home/mturner /Users/matthewturner/Documents/Cluster_backup/

echo "Done"

The rsync flags are described below:

• –delete: delete extraneous files from dest dirs
• -a: Archive mode
• -v: Increase Verbosity
• -z: Compress file data during the transfer
• -e: specify the remote shell to use