Backup with rsync using SSH Tunneling

For those of you that read my blog often, you know that I admin the cluster that our research group uses here at CU Boulder.  Because of this, I get a lot of questions from users who don’t want to take the time to solve their own problems.  Fairly recently, our RAID-6 crashed (we had a 4th drive die and had to rebuild the array).  Normally this wouldn’t be very much of a problem as most of the files saved on our storage drive are just input files that we can re-download from a separate server, or so I thought.  Personally, all my source code is in my home folder, backed up on our data server, and backed up onto my personal laptop.  For researchers in our group who are developing code, not having a backup of source code can lead to many many months of lost work.  Well, as it turns out, many of the people in our group had their source code on our data server (the one that crashed), without a backup anywhere.  So months of work had been lost.  Well, after the rebuild I have gotten many questions on how to set up an ssh tunnel so that they can backup from our cluster, through the front end, to their home computer.

Anyway, here is how I was able to setup a passthrough on my system.  It should be noted that I have enabled password-less SSH to the frontend and from the frontend to the server.

ssh -fN -L 2222::22 @

# Transfer from cluster to local
scp -o"Port=2222" @localhost:/path/on/cluster/ /path/on/personal/computer/
# Transfer from local to prospero
scp -o"Port=2222" /path/on/personal/computer/ @localhost:/path/on/cluster/

In the above ssh command, the flags are as follows:

  • -f: Requests ssh to go to background just before command execution
  • -N: Do not execute a remote command
  • -L [port:host:host port]: Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side

Example: (This is a purely artificial scenario, as there is no server cluster.mattdturner.com and there would be no need to go through google.com) If you want to access a cluster (cluster.mattdturner.com) but you can only access the server by first accessing a frontend (google.com), you would use the following command.

ssh -fN -L 2222:cluster.mattdturner.com:22 username@google.com

Then, to download a file (text.file) in your home directory on cluster.mattdturner.com to your local home directory,

scp -o"Port=2222" username@localhost:~/text.file ~/

Now that you are familiar with SSH Tunneling, lets setup an rsync to utilize the SSH tunnel to backup certain files from a remote server. In my case, I setup a c-shell script that is run each night at midnight to backup all of my source code. The script is below:

#!/bin/csh

date

rsync --delete -avz -e "ssh mturner@ ssh" mturner@:/home/mturner /Users/matthewturner/Documents/Cluster_backup/

echo "Done"

The rsync flags are described below:

  • –delete: delete extraneous files from dest dirs
  • -a: Archive mode
  • -v: Increase Verbosity
  • -z: Compress file data during the transfer
  • -e: specify the remote shell to use
And then just add the execution of the script to cron.
Any questions or comments?  Leave them below and I will respond in a timely fashion.
Also, if anybody knows how to enable wrapping text with the WordPress Plugin SyntaxHighlighter Evolved, could you please leave that information in the comments.  Thanks!
Enhanced by Zemanta

8 thoughts on “Backup with rsync using SSH Tunneling

  1. Timothy Deakins

    Do you mind if I quote a few of your posts as long as I provide credit and sources back to your weblog? My website is in the exact same niche as yours and my visitors would genuinely benefit from some of the information you provide here. Please let me know if this ok with you. Thanks a lot!|

    • necktweaker

      Feel free to quote any material from any of the posts on this site. Thanks for dropping by 🙂

  2. For centuries men have kept an appointment with Christmas. Christmas means fellowship, feasting, giving and receiving, a time of good cheer, home. ~W.J. Ronald Tucker

  3. I went more than this site and I conceive you have a great deal of great info, saved to fav (:.

  4. What a good post. I read this post many time and enjoyed a lot and got very good information here. I hope i will visit this post in coming future as well. safety flooring

  5. I have never seen this type of best article, thanks for sharing here.

  6. Thanks for sharing this great post.

  7. It's one the best posts, thanks for this great work.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> 

This site uses Akismet to reduce spam. Learn how your comment data is processed.