{"id":369,"date":"2012-09-28T10:27:52","date_gmt":"2012-09-28T16:27:52","guid":{"rendered":"http:\/\/mattdturner.com\/wordpress\/?p=369"},"modified":"2012-09-28T10:50:37","modified_gmt":"2012-09-28T16:50:37","slug":"backup-with-rsync-using-ssh-tunneling","status":"publish","type":"post","link":"http:\/\/mattdturner.com\/wordpress\/2012\/09\/backup-with-rsync-using-ssh-tunneling\/","title":{"rendered":"Backup with rsync using SSH Tunneling"},"content":{"rendered":"<p style=\"text-align: center;\"><a href=\"http:\/\/mattdturner.com\/wordpress\/wp-content\/uploads\/2012\/09\/SSH_Tunnel.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter  wp-image-435\" title=\"SSH_Tunnel\" src=\"http:\/\/mattdturner.com\/wordpress\/wp-content\/uploads\/2012\/09\/SSH_Tunnel-1024x408.jpg\" alt=\"\" width=\"516\" height=\"206\" srcset=\"http:\/\/mattdturner.com\/wordpress\/wp-content\/uploads\/2012\/09\/SSH_Tunnel-1024x408.jpg 1024w, http:\/\/mattdturner.com\/wordpress\/wp-content\/uploads\/2012\/09\/SSH_Tunnel-300x119.jpg 300w, http:\/\/mattdturner.com\/wordpress\/wp-content\/uploads\/2012\/09\/SSH_Tunnel-768x306.jpg 768w, http:\/\/mattdturner.com\/wordpress\/wp-content\/uploads\/2012\/09\/SSH_Tunnel.jpg 1906w\" sizes=\"(max-width: 516px) 100vw, 516px\" \/><\/a><\/p>\n<p style=\"text-align: left;\">For those of you that read my blog often, you know that I admin the cluster that our research group uses here at CU Boulder. \u00a0Because of this, I get a lot of questions from users who don&#8217;t want to take the time to solve their own problems. \u00a0Fairly recently, our RAID-6 crashed (we had a 4th drive die and had to rebuild the array). \u00a0Normally this wouldn&#8217;t be very much of a problem as most of the files saved on our storage drive are just input files that we can re-download from a separate server, or so I thought. \u00a0Personally, all my source code is in my home folder, backed up on our data server, and backed up onto my personal laptop. \u00a0For researchers in our group who are developing code, not having a backup of source code can lead to many many months of lost work. \u00a0Well, as it turns out, many of the people in our group had their source code on our data server (the one that crashed), without a backup anywhere. \u00a0So months of work had been lost. \u00a0Well, after the rebuild I have gotten many questions on how to set up an ssh tunnel so that they can backup from our cluster, through the front end, to their home computer.<\/p>\n<p><!--more-->Anyway, here is how I was able to setup a passthrough on my system. \u00a0It should be noted that I have enabled password-less SSH to the frontend and from the frontend to the server.<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\nssh -fN -L 2222::22 @\r\n\r\n# Transfer from cluster to local\r\nscp -o&quot;Port=2222&quot; @localhost:\/path\/on\/cluster\/ \/path\/on\/personal\/computer\/\r\n# Transfer from local to prospero\r\nscp -o&quot;Port=2222&quot; \/path\/on\/personal\/computer\/ @localhost:\/path\/on\/cluster\/\r\n<\/pre>\n<p>In the above ssh command, the flags are as follows:<\/p>\n<ul>\n<li><strong>-f<\/strong>:\u00a0Requests ssh to go to background just before command execution<\/li>\n<li><strong>-N<\/strong>:\u00a0Do not execute a remote command<\/li>\n<li><strong>-L [port:host:host port]<\/strong>:\u00a0Specifies that the given port on the local (client) host is to be\u00a0forwarded to the given host and port on the remote side<\/li>\n<\/ul>\n<p><strong>Example<\/strong>: (This is a purely artificial scenario, as there is no server cluster.mattdturner.com and there would be no need to go through google.com) If you want to access a cluster (cluster.mattdturner.com) but you can only access the server by first accessing a frontend (google.com), you would use the following command.<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">ssh -fN -L 2222:cluster.mattdturner.com:22 username@google.com<\/pre>\n<p>Then, to download a file (text.file) in your home directory on cluster.mattdturner.com to your local home directory,<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">scp -o&quot;Port=2222&quot; username@localhost:~\/text.file ~\/<\/pre>\n<p>Now that you are familiar with SSH Tunneling, lets setup an rsync to utilize the SSH tunnel to backup certain files from a remote server. In my case, I setup a c-shell script that is run each night at midnight to backup all of my source code. The script is below:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n#!\/bin\/csh\r\n\r\ndate\r\n\r\nrsync --delete -avz -e &quot;ssh mturner@ ssh&quot; mturner@:\/home\/mturner \/Users\/matthewturner\/Documents\/Cluster_backup\/\r\n\r\necho &quot;Done&quot;<\/pre>\n<p>The rsync flags are described below:<\/p>\n<ul>\n<li><strong>&#8211;delete<\/strong>:\u00a0delete extraneous files from dest dirs<\/li>\n<li><strong>-a<\/strong>: Archive mode<\/li>\n<li><strong>-v<\/strong>: Increase Verbosity<\/li>\n<li><strong>-z<\/strong>:\u00a0Compress file data during the transfer<\/li>\n<li><strong>-e<\/strong>:\u00a0specify the remote shell to use<\/li>\n<\/ul>\n<div>And then just add the execution of the script to cron.<\/div>\n<div><\/div>\n<div>Any questions or comments? \u00a0Leave them below and I will respond in a timely fashion.<\/div>\n<div><\/div>\n<div>Also, if anybody knows how to enable wrapping text with the <a href=\"http:\/\/wordpress.org\/extend\/plugins\/syntaxhighlighter\/\">WordPress Plugin SyntaxHighlighter Evolved<\/a>, could you please leave that information in the comments. \u00a0Thanks!<\/div>\n<div class=\"zemanta-pixie\" style=\"margin-top: 10px; height: 15px;\"><a class=\"zemanta-pixie-a\" title=\"Enhanced by Zemanta\" href=\"http:\/\/www.zemanta.com\/?px\"><img decoding=\"async\" class=\"zemanta-pixie-img\" style=\"border: none; float: right;\" src=\"http:\/\/img.zemanta.com\/zemified_e.png?x-id=7c85f222-518f-4fdd-9188-9ced85c361fb\" alt=\"Enhanced by Zemanta\" \/><\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>For those of you that read my blog often, you know that I admin the cluster that our research group uses here at CU Boulder. \u00a0Because of this, I get a lot of questions from users who don&#8217;t want to take the time to solve their own problems. \u00a0Fairly recently, our RAID-6 crashed (we had [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":435,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[3,64,11,20],"tags":[162,232,231,127,230,142,229,48],"class_list":["post-369","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-command-line","category-linux-2","category-mac","category-scripting","tag-bash","tag-clients","tag-protocols","tag-secure-shell","tag-server-computing","tag-ssh","tag-tunneling-protocol","tag-unix"],"_links":{"self":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts\/369"}],"collection":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/comments?post=369"}],"version-history":[{"count":11,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts\/369\/revisions"}],"predecessor-version":[{"id":429,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/posts\/369\/revisions\/429"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/media\/435"}],"wp:attachment":[{"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/media?parent=369"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/categories?post=369"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/mattdturner.com\/wordpress\/wp-json\/wp\/v2\/tags?post=369"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}