CCP14 Homepage - Single Crystal and Powder Diffraction - CCP14 Config Administration Information - Server Security Information - Routine, flexible and convenient Windows to UNIX Web updating using Rsync

[CCP14 Home: (Frames \| No Frames)] CCP14 Mirrors: [UK] \| [CA] \| [US] \| [AU]	What's New	Introduction	Site Map
Search the CCP14	Download Programs What do you want to do? (lists of software by crystallographic method)	Tutorials	Solutions

(This Webpage Page in No Frames Mode)

Collaborative Computational Project Number 14

for Single Crystal and Powder Diffraction

CCP14

Server Security Information

Routine, flexible and convenient Windows to UNIX Web updating using Rsync

The CCP14 Homepage is at http://www.ccp14.ac.uk

[Back to CCP14 Web/Config Main Page]

[Security Links Homepage] | [To FTP secure shell Tunnelling Page] | [To X secure shell Tunnelling] | [Routine Windows to UNIX Web updating using Rsync] | [Secure Routine Windows to UNIX Web updating using Teraterm and Rsync]

What the point of this?

The point of this is to be able do very easy web updating (even over tortuously slow phone lines) in a convenient, timely and effective manner. FTP can be quite painful in this regard but Rsync can be superb. If you change a line of a 500kB ASCII file, rsync will only need to send around 2kB of the "changes" over - and it will compress, and run via secure shell if desired.

The following is based around the excellent "Installing ssh and rsync on a Windows machine" by Robert Scholten at http://optics.ph.unimelb.edu.au/help/rsync/

However, a big difference is that this does not go via secure shell as I could not get the default compiled ssh to work. The following tutorial on how to "tunnel" via "Teraterm/ttsh Secure Shell" to the rsync deamon could be more convenient: Secure Routine Windows to UNIX Web updating using Teraterm and Rsync

Notes:

Also, rsync can be quite memory and CPU intensive with large file systems, so it can be necessary sometimes to split a big rsyncing into bits. (e.g., trying to do the entire CCP14 area at once (9Gig from the PC, it crashes PC - looks like too many files. Though this is also possibly due to nuances in the Windows compilation of the latest Rsync (April 2000))
The first time you run this rsync method, it may want to "touch" the files: most likely because they differ in a minor way in the "file times" and in their ASCII formats. So the DOS ASCII file format will be transferred to the UNIX machine - which does not cause a problem with Apache.
Rsync has mega functionality. Including automatically making backups of uploaded files, etc. Read the documentation if you think some of these features may be good for what you want to do.

Relevant pages:

Rsync Homepage: http://rsync.samba.org
CCP14 Administration webpages: Rsync software for Co-operative Incremental and full Mirroring: http://www.ccp14.ac.uk/ccp14admin/rsync/

Install and configure Rsync on your UNIX (VMS?) webserver. You can use the following information.

CCP14 Administration webpages: Rsync software for Co-operative Incremental and full Mirroring: http://www.ccp14.ac.uk/ccp14admin/rsync/

Methods may vary, but put in a user specific config in the /etc/rsyncd.conf file. One for each user.

motd file = /etc/motd max connections = 25 syslog facility = local3 # Normally you would set read only = yes - but we want the user to be # able to write files to his area. [joe_blogs_web] comment = updating Joe Blogs's webpage via rsync path = /path/to/web_area/for/joe read only = no list = yes uid = real_user_id gid = real_user_group hosts allow = machine1.dl.ac.uk machine2.domain.com machine2.dl.ac.uk auth users = rsync_specific_username secrets file = /etc/rsyncd.secrets

NOTE: If you change the rsyncd.secrets file, then (in inetd mode) REHUP Inetd on the server using /etc/killall -HUP inetd to get this changed noticed. Otherwise, you may get an auth error, even if you are inserting the correct rsync password.

Downloading and installing rsync for Windows

Look at the instructions at Robert Scholten's "Installing ssh and rsync on a Windows machine" by Robert Scholten at http://optics.ph.unimelb.edu.au/help/rsync/

Download the rsync/ssh package from http://optics.ph.unimelb.edu.au/help/rsync/rsync_package.zip

Extra the zip so that it creates subdirectories in the c:\etc and c:\rsync area. This is important if you are trying to use the secureshell option that comes with this distribution of "rsync for Windows". If you extract these into other directories such as c:\rsync\etc and c:\rsync\sync, the following secure shell part will not work. This is to take into account some UNIX nuances of this particular Windows implementation of rsync.

In the c:\rsync directory, rename one of the rsync_version.exe to rsync.exe. At this date, I used the rsync2.4.3.exe version. If you get problems in file transfers, you may have to try another version.

ren rsync2.4.3.exe rsync.exe

Setting up rsync command file

The method to use to start with is to always try this with --dry-run option first to see what will happen. It is possible to do nasty things with rsync - it will obey what you tell it to do (which may not be what you think you requested). Also, don't forget to turn off the --dry-run option when you want the files to actually transfer across.

Another hint is don't use the --delete option. One tired move with this option and you may accidentally delete a portion of the web-area that you did not mean to. If you need to elliminate files, it might be best to do this using a GUI FTP program.

Following is an example DOS style bat batch file to run various options depending on what is required. (it has a lot of the common tools "REMMED" ready for cut and paste) Once you get into a routine, you may create you own discrete batch files. If doing a set of work on a certain directory system, extend a command line for that directory. --exclude "*.pfe" is the backup file name that the freeware PFE for Windows text editor creates when editing the HTM and HTML file. So we don't want to transfer the backup files over. The "\\d/" stands for D: drive on the hard-disk.

We specify the rsync area we want to update (in this case, joe_blogs_web), the rsync specific user name, and the option.

@echo off
REM c:\rsync\rsync -av --delete "\\d/" torte:/rsync/data2/scholten/schizo/d >>c:\rsync\rsync.log
REM netjam --progress --safe-links --dry-run
REM http://optics.ph.unimelb.edu.au/help/rsync/#install
REM http://www.ccp14.ac.uk/ccp14admin/rsync/index.html --exclude "*.gif" --exclude "*.jpg"
set CYGWIN=tty binmode
set TERM=ansi
REM set RSYNC_PASSWORD=plaintextpassword
REM >>c:\rsync\rsync.log
set RSYNC_RSH=c:\rsync\ssh.exe
REM echo Drive D >>c:\rsync\rsync.log

c:\rsync\gnudate

REM Books test
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --delete --stats -e ssh -z "\\d/lachlan/web/books/" [email protected]_server.ac.uk::joe_blogs_web/books/

REM root area HTML changes (what's new). Currently running
c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --exclude "*.gif" --exclude "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" [email protected]_server.ac.uk::joe_blogs_web/

REM Images test
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/images/" [email protected]_server.ac.uk::joe_blogs_web/images/

REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" [email protected]_server.ac.uk::joe_blogs_web/

REM Complete update
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --stats -e ssh -z "\\d/lachlan/web/" [email protected]_server.ac.uk::joe_blogs_web/

REM Root area and tutorial area
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/tutorial/" --include "/tutorial/**" --exclude "**" --stats -e ssh -z "\\c/lachlan/ccp14/web_live/" [email protected]_server.ac.uk::joe_blogs_web/

REM Root area and tutorial directory and no subdirectories
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/tutorial/" --exclude "**" --stats -e ssh -z "\\c/lachlan/ccp14/web_live/" [email protected]_server.ac.uk::joe_blogs_web/

REM ccp14admin security area + rsync + ccp14admin index page
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.txt" --include "*.zip" --include "*gz" --include "*Z" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/rsync/" --include "/rsync/**" --include "/security/" --include "/security/**" --exclude "**" --stats -z "\\c/lachlan/ccp14/web_live/ccp14admin/" [email protected]_server.ac.uk::joe_blogs_web/ccp14admin/

c:\rsync\gnudate

Running Rsync

In this example, we have modified a file in the root web-area, whatsnew.htm. So the DOS style BATCH program we run has an rsync command line that only updates the HTML files in the root web-area, and not in any subdirectory.

c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --exclude "*.gif" --exclude "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" [email protected]_server.ac.uk::joe_blogs_web/

Rsync will then prompt for the "rsync" password corresponding to this username and the transfer will continue.

You can avoid having to type in a password by using the following declaration in your batch file (but this is not a good habit to get into):

set RSYNC_PASSWORD=plaintextpassword

In the following screen dump, rsync has started, prompted for the password (which the user enters); then determins what has changed, and updates the file. Note that even though the whatsnew.htm file is 47 kB, rsync only had to transfer 808 bytes to update the remote file on the web server to match the local version. This means it takes far less time and bandwidth.

Rsync running

To repeat the uploading exercise, have a DOS window open with doskey installed, and just press the "up arrow" key and press enter to run the previous command.

[Back to CCP14 Web/Config Main Page]

[CCP14 Home: (Frames \| No Frames)] CCP14 Mirrors: [UK] \| [CA] \| [US] \| [AU]	What's New	Introduction	Site Map
Search the CCP14	Download Programs What do you want to do? (lists of software by crystallographic method)	Tutorials	Solutions

(This Webpage Page in No Frames Mode)

If you have any queries or comments, please feel free to contact the CCP14