Thus:
One work of warning is that the present set of rsync programs (1998 to Jan 2000 and beyond?) can be a bit finicky when it comes to transferring large chunks of data. Possible due to memory requirements(?). Thus it can be better to split the mirroring into smaller sections. While a pain, this gets the job done.
Relevant pages:
Also, it is possible that if you are getting the following type of problem.
this can be due to compiling up rsync with GCC on IRIX. Switch to
CC and things should be OK.
cu11 87% ./rsync.ccp14.nrc
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
With the corresponding error in the Syslog file on the server
(this is running in deamon mode)
Jan 29 02:18:27 6T:sv1 rsyncd[1803123]: transfer
interrupted (code 12) at io.c(167)
|
There is an intension to implement a more effecient incremental system for backups going into the rsync program.
Also, note that the method for doing excludes changed with 2.3.0 such that a ** is required to recurse all subdirectories.
As noted before, GCC can give problems with IRIX so it is best to use the IRIX cc compiler.
From: [[email protected]] To: Multiple recipients of list RSYNC [[email protected]] Subject: rsync 2.4.0 released X-URL: http://lists.samba.org/ Date: Sat, 29 Jan 2000 23:34:21 +1100 I've just released rsync 2.4.0. The number of test reports I got tailed off and none of them were negative so I decided to do a release. See http://rsync.samba.org/ for the release notes and source. Note that to _really_ fix the problem with rsync over ssh you'll also have to compile sshd to use socketpair() instead of pipe(). I have a little demo program that shows that sshd can definately lock up under bi-directional transfers when using pipe(). I believe that is the primary cause of rsync over ssh problems. To compile openssh with socketpair instead of pipes, just remove the line: #define USE_PIPES 1 from near the bottom of includes.h Cheers, Tridge
Refer down to the bottom of this page for hints on using ssh/Secure Shell properly with rsync |
rsync 873/tcp # Rsync Deamon for mirroring
rsync stream tcp nowait root /usr/local/bin/rsync rsyncd --daemon
motd file = /etc/motd max connections = 25 syslog facility = local3 [ccp14] comment = ccp14 regional mirror area ignore nonreadable = yes timeout = 600 path = /web_disc/ccp14/web_area read only = yes list = yes uid = nobody gid = nobody hosts allow = *.doobry.ac.uk 148.79.*.* auth users = rsyncman secrets file = /etc/rsyncd.secrets
Note:
> All parameters are in parameter/value pairs, joined by '=' characters. > This is important even for apparent simple assertions, as there is only > one name for each parameter... i.e. there is no "do not ignore > nonreadable", or "do not use chroot", but rather "ignore nonreadable = no" > and "use chroot = no". > > ignore nonreadable = yes > timeout = 600 > > Tim Conway
rsyncman:plaintextpassword
Then as root, use the chmod 600 rsyncd.secrets command to set the rsyncd.secrets file as read/write only by root.
local3.info /var/adm/SYSLOG
ignore errors: The "ignore errors" option tells rsyncd to ignore IO errors on the server when deciding whether to run the delete phase of the transfer. Normally rsync skips the --delete step if any IO errors have occurred in order to prevent disasterous deletion due to a temporary resource shortage or other IO error. In some cases this test is counter productive so you can use this option to turn off this behaviour.
ignore nonreadable: This tells the rsync server to completely ignore files that are not readable by the user. This is useful for public archives that may have some non-readable files
A previous version of this web page used the -a (archive - equivalent to using -rlptDg) option. Be wary that using the -a archive option sets the -g (group flag) for files to have the same group as that on the remote system. This is not necessarily what you want to have happen when you are mirroring foreign web and ftp sites. Use -rlpt flag instead combined with the --safe-links flag to handle symbolic links.
#!/bin/csh setenv RSYNC_PASSWORD plaintextpassword rsync -rlpt -v --stats -e ssh -z --safe-links --delete \ --exclude=ccp/ccp14/ftp-mirror/alife --exclude=ccp/ccp14/ftp-mirror/programming \ [email protected]::ccp14/web_live/ ./web_area/web_live/ > log.rsync.txt # rsync -avz rsync://rsync.samba.org/ftp/unpacked/rsync/ rsync-2.3.3pre2 # --include "/[a-f]*/" \ # --include "/[a-f]*/**" \ # -n, --dry-run # --progress rsync -rlpt -v --stats -e ssh -z --safe-links --delete \ [email protected]::ccp14/programming/ftp-mirror/programming/ \ ./web_area/web_live/ccp/ccp14/ftp-mirror/programming/ >> log.rsync.txt rsync -rlpt -v --stats -e ssh -z --safe-links --delete \ [email protected]::ccp14/programming/ftp-mirror/mumit-khan/ \ ./web_area/web_live/ccp/ccp14/ftp-mirror/programming/mumit-khan/ >> log.rsync.txt rsync -rlpt -v --stats -e ssh -z --safe-links --delete \ [email protected]::ccp14/alife/ftp-mirror/alife/ \ ./web_area/web_live/ccp/ccp14/ftp-mirror/alife/ >> log.rsync.txt
--include "/[a-f]*/" \
--include "/[a-f]*/**" \
In this case, the username(rsyncmir), server machine name(www.ccp14.ac.uk), profile name(ccp14) and directory area to mirror from(/web_live/) and to(./web_area/web_live/).
Example Partial CCP14 Mirroring Script(Where the mirror server does not room for the entire mirror)#!/sbin/csh #!/bin/bash #Secure copying updates to the remote clients #scp rsync.ccp14.nrc [email protected]:/usr/people/guest/ccp14mirroring/rsync.c cp14.nrc #scp report_template_nrc.txt [email protected]:/usr/people/guest/ccp14mirroring /report_template_nrc.txt # scp rsync.ccp14 [email protected]:/usr/local/sys/rsync.ccp14 # You should CHANGE THE NEXT 3 LINES to suit your local setup setenv CCP14WEBDIR /usr/people/guest/web_area/ccp14 setenv REPORTDIR /usr/people/guest/ccp14mirroring setenv RSYNC_PASSWORD insert_password_here # export RSYNC_PASSWORD=insert_password_here # original script kept on sv1.ccp14.ac.uk # /web_disc/ccp14/cron_scripts/remote #rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ cp $REPORTDIR/report_template_nrc.txt $REPORTDIR/report.txt set DATE=(`date`) # DATE=`date` # bash style # -v verbose # -q quiet # --dry-run # --delete-after delete old file after downloading the update sed "/Mirror_of_CCP14_Started/s/NOT_FINISHED/ $DATE/" \ $REPORTDIR/report.txt > $REPORTDIR/report.txt.new mv $REPORTDIR/report.txt.new $REPORTDIR/report.txt /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/ansto/" \ --include "/ansto/**" \ --include "/axes/" \ --include "/axes/**" \ --include "/briantoby/" \ --include "/briantoby/**" \ --include "/crush/" \ --include "/crush/**" \ --include "/crystals/" \ --include "/crystals/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/ ccp14/ftp-mirror/ > $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/debvin/" \ --include "/debvin/**" \ --include "/dirdif/" \ --include "/fit2d/**" \ --include "/fit2d/" \ --include "/dirdif/**" \ --include "/fullprof/" \ --include "/fullprof/**" \ --include "/gsas/" \ --include "/gsas/**" \ --include "/jana/" \ --include "/jana/**" \ --include "/krumm/" \ --include "/krumm/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/ ccp14/ftp-mirror/ >> $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/ill-hewat/" \ --include "/ill-hewat/**" \ --include "/lauecell-ravelli/" \ --include "/lauecell-ravelli/**" \ --include "/lapod-langford/" \ --include "/lapod-langford/**" \ --include "/llnlrupp/" \ --include "/llnlrupp/**" \ --include "/lutterotti/" \ --include "/lutterotti/**" \ --include "/nardelli/" \ --include "/nardelli/**" \ --include "/ornl-ortep/" \ --include "/ornl-ortep/**" \ --include "/platon-spek/" \ --include "/platon-spek/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/ ccp14/ftp-mirror/ >> $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/powdcell/" \ --include "/powdcell/**" \ --include "/profil/" \ --include "/profil/**" \ --include "/prometheus/" \ --include "/prometheus/**" \ --include "/rietan/" \ --include "/rietan/**" \ --include "/weber/" \ --include "/weber/**" \ --include "/wget/" \ --include "/wget/**" \ --include "/xnd/" \ --include "/xnd/**" \ --include "/xray-villers/" \ --include "/xray-villers/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/ ccp14/ftp-mirror/ > $REPORTDIR/Log.ccp14 set DATE=(`date`) # DATE=`date` # bash style sed "/FTP_mirror/s/NOT_FINISHED/Finished $DATE/" $REPORTDIR/report.txt > $REPORT DIR/report.txt.new mv $REPORTDIR/report.txt.new $REPORTDIR/report.txt /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/armel/" \ --include "/armel/**" \ --include "/bgmn/" \ --include "/bgmn/**" \ --include "/briantoby/" \ --include "/briantoby/**" \ --include "/caos/" \ --include "/caos/**" \ --include "/crush/" \ --include "/crush/**" \ --include "/crys-r-shirley/" \ --include "/crys-r-shirley/**" \ --include "/crystals/" \ --include "/crystals/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m irrors/ >> $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/dirdif/" \ --include "/dirdif/**" \ --include "/dplot/" \ --include "/dplot/**" \ --include "/fit2d/**" \ --include "/fit2d/" \ --include "/ethz/" \ --include "/ethz/**" \ --include "/farrugia/" \ --include "/farrugia/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m irrors/ >> $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/hugorietveld/" \ --include "/hugorietveld/**" \ --include "/jana/" \ --include "/jana/**" \ --include "/krumm/" \ --include "/krumm/**" \ --include "/ill-hewat/" \ --include "/ill-hewat/**" \ --include "/lapod-langford/" \ --include "/lapod-langford/**" \ --include "/llnlrupp/" \ --include "/llnlrupp/**" \ --include "/lutterotti/" \ --include "/lutterotti/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m irrors/ >> $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/nardelli/" \ --include "/nardelli/**" \ --include "/ndragoe/" \ --include "/ndragoe/**" \ --include "/ornl-ortep/" \ --include "/ornl-ortep/**" \ --include "/ortex/" \ --include "/ortex/**" \ --include "/patsee/" \ --include "/patsee/**" \ --include "/pfe/" \ --include "/pfe/**" \ --include "/pki/" \ --include "/pki/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m irrors/ >> $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/platon-spek/" \ --include "/platon-spek/**" \ --include "/plotr/" \ --include "/plotr/**" \ --include "/powdcell/" \ --include "/powdcell/**" \ --include "/rietan/" \ --include "/rietan/**" \ --include "/valence/" \ --include "/valence/**" \ --include "/weber/" \ --include "/weber/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m irrors/ >> $REPORTDIR/Log.ccp14 /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ --include "/winmprof/" \ --include "/winmprof/**" \ --include "/wget/" \ --include "/wget/**" \ --include "/xnd/" \ --include "/xnd/**" \ --include "/xray-villers/" \ --include "/xray-villers/**" \ --include "/xtalview-mcree/" \ --include "/xtalview-mcree/**" \ --exclude "*" \ [email protected]::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m irrors/ >> $REPORTDIR/Log.ccp14 set DATE=(`date`) # DATE=`date` # bash style sed "/Web_Mirror/s/NOT_FINISHED/Finished $DATE/" $REPORTDIR/report.txt > $REPORT DIR/report.txt.new mv $REPORTDIR/report.txt.new $REPORTDIR/report.txt /usr/local/bin/rsync -rlpt -e ssh -v --stats -z --safe-links --delete \ [email protected]::ccp14/nrc/ $CCP14WEBDIR/nrc/ >> $REPORTDIR/Log.ccp14 set DATE=(`date`) # DATE=`date` # bash style sed "/HTML_Index_files/s/NOT_FINISHED/Finished $DATE/" $REPORTDIR/report.txt > $ REPORTDIR/report.txt.new mv $REPORTDIR/report.txt.new $REPORTDIR/report.txt /usr/sbin/Mail -s "AECL_daily_rsync_mirroring_Results `date`" [email protected] < / usr/people/guest/ccp14mirroring/report.txt # /usr/ucb/Mail -s "CSIRO_daily_rsync_mirroring_Results `date`" [email protected] < /usr/people/guest/ccp14mirroring/report.txt # -n, --dry-run # --progress |
20 5 * * * ./rsync.script
# 3rd October 2001 - Lachlan Cranswick - autoscripts for CSIRO CCP14 mirroring # ([email protected] - http://www.ccp14.ac.uk) # rsync mirroring starts a few hours after CCP14 mirroring has finished # http://www.timeanddate.com/worldclock/ # Main mirroring at ~6:20am UK time, then updates at # ~2:20pm, ~5:20pm, ~8:20pm, ~11:20pm UK time # Make sure last rsync script is dead by 12:07am as # CCP14 UK wget mirroring starts 21 17 * * * ./ccp14mirroring/rsync.ccp14.csiro 14 1,4,7,10 * * * /usr/sbin/killall -9 rsync.ccp14.csiro 16 1,4,7,10 * * * /usr/sbin/killall -9 rsync 18 1,4,7,10 * * * /usr/sbin/killall -9 rsync 21 1,4,7,10 * * * ./ccp14mirroring/rsync.ccp14.csiro
Starting up Rsync so that it only binds to one IP addressFrom: [[email protected]] To: Lachlan Cranswick [[email protected]] Cc: [email protected] Subject: Re: rsync: network "bind interfaces only" command for rsync? Date: Tue, 14 May 2002 07:44:37 -0600 Might the --address= option below, in conjunction with your IP filtering, do what you want? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ --address By default rsync will bind to the wildcard address when run as a daemon with the --daemon option or when con- necting to a rsync server. The --address option allows you to specify a specific IP address (or hostname) to bind to. This makes virtual hosting possible in con- junction with the --config option. SunOS 5.7 Last change: 25 Jan 2002 17 User Commands rsync(1) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Tim Conway [email protected] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(nnnnnnnnnnnn, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), ".\n" ' "There are some who call me.... Tim?" |
Possibilities for the Future - Ultra High Effeciency MirroringFrom the Rsync Mailing List:
Forwarded message: >From [email protected] Wed Dec 9 00:38:15 1998 Message-Id: >[email protected]< Errors-To: [email protected] Reply-To: [email protected] Originator: [email protected] Sender: [email protected] Precedence: bulk From: Andrew Tridgell >[email protected]< To: Multiple recipients of list >[email protected]< Subject: file list cache X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas X-URL: http://samba.anu.edu.au/listproc X-Comment: Discussion of rsync package Date: Wed, 9 Dec 1998 11:37:43 +1100 While writing up part of rsync for my thesis it occurred to me that we can improve the mirroring performance considerably by cacheing the file list on the client. It is quite common that the file list is the most network intensive part of a rsync mirror run, often making up over 90% of the bytes on the wire. It is usually about 10-14 bytes per file so if you are mirroring large numbers of files (say millions) it can soon grow to be a significant network cost. I think the solution is a --cache option that works like this: - the first time it is used the current behaviour is unchanged, except that the client saves the file list as a single file in /tmp/rsync-cache-$USER/XXXXX.cache where XXXXX is a hash of the commandline options. (obviously the directory would need to be able to be chosen with an option as well, just defaulting to this) - the next time through rsync detects this file and the client and server use the rsync algorithm to update the cache to the correct file list. The idea is that the file list won't change much between runs so it is an ideal candidate for rsync. The client sends the checksum signatures of the current cache file to the server and the server sends back the rsync differences to bring the cache up to date. The nice thing about this system is that it avoids the network traffic associated with transferring the file list in most cases yet it guarantees that the client has an accurate file list. If we combine this with an option to create/use a file list in a file on the server then I think many of the current mirroring issues will be solved. This will require quite a bit of work in the code and probably won't happen till next year, but I thought I'd mention it now. I thought the idea of using rsync to update the rsync file lists is quite nice :)
Rsync for Windows (including binaries)Installing ssh and rsync on a Windows machine
http://optics.ph.unimelb.edu.au/help/rsync/
Compiled with cygwin-b20. -- Robert Scholten Tel: +61 3 9344 5457 School of Physics Fax: +61 3 9347 4783 University of Melbourne email: [email protected] Victoria 3010 AUSTRALIA http://www.ph.unimelb.edu.au/~scholten/ How to setup Rsync as an NT Service: http://members.home.net/cbollerud2/projects/rsync/NTService.html
From: JF Paradis <[email protected]> To: [email protected] Subject: Re: Another way to setup Rsync as an NT Service Date: Thu, 22 Nov 2001 13:09:18 -0500 Here is how: Use the KILL.EXE from the NT Resource kit. On my systems, I use the NT scheduler to run batch files (using the default AT or the GUI WINAT.EXE from the NT resource kit). At the start of each batch file, I use: c:\bin\KILL.EXE rsync This ensure that rsync is not already running! Here is my setup: 1. No need to install cygwin 2. I have FS1 & FS2 to backup on FSX 3. The same client batch file is used (use %COMPTUERNAME%) 4. I backup the client e: drive to the server .bak folders Put into c:\bin of all machines: - rsync.exe - cygwin1.dll - kill.exe - srvany.exe - instsrv.exe Along with these text files (copy everything between the tags): <rsync.bat> SET OPTION=--compress --recursive --times --delete --bwlimit=75 SET SELECT=--exclude "*bak" --exclude "*~" SET SOURCE=/cygdrive/e/ SET SERVER=FSX::%COMPUTERNAME% c:\bin\kill.exe rsync c:\bin\rsync.exe %OPTION% %SELECT% %SOURCE% %SERVER% <EOF> <rsybcd.conf> use chroot = false strict modes = false read only = false [FS1] path = /cygdrive/e/fs1.bak hosts allow = fs1 [FS2] path = /cygdrive/e/fs2.bak hosts allow = fs2 <EOF> <rsyncd_install.bat> c:\bin\instsrv.exe rsyncd "c:\bin\srvany.exe" c:\bin\rsyncd.reg pause <EOF> <rsyncd_remove.bat> net stop rsyncd c:\bin\kill.exe rsync c:\bin\instsrv.exe rsyncd remove pause <EOF> <rsyncd_restart.bat> net stop rsyncd c:\bin\kill.exe rsync net start rsyncd pause <EOF> <rsyncd_stop.bat> net stop rsyncd c:\bin\kill.exe rsync pause <EOF> <rsyncd.reg> REGEDIT4 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\rsyncd] [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\rsyncd\Parameters] "AppDirectory"="c:\\program files\\rsync" "Application"="c:\\program files\\rsync\\rsync.exe" "AppParameters"="--daemon --config=rsyncd.conf" <EOF> <rsyncd_schedule.bat> at \\fs1 10:00pm /every:m,t,w,th,f,s,su c:\bin\rsync.bat at \\fs2 11:00pm /every:m,t,w,th,f,s,su c:\bin\rsync.bat <EOF> Then double-click on thse for the initial setup: rsync_install.bat rsync_restart.bat rsync_schedule.bat Use these to manage the service: rsync_stop.bat rsync_restart.bat Use this to uninstall: rsync_remove.bat Easy, simple and clean! JF |
Rsync for Windows NT running in Deamon ModeRefer: http://rsync.samba.org/rsync/nt.htmlSender: [email protected] From: "Mike McHenry" [[email protected]] To: Multiple recipients of list RSYNC [[email protected]] Subject: Rsync 2.3.1 WinNT binaries and instructions available X-URL: http://samba.anu.edu.au/listproc Date: Fri, 15 Oct 1999 02:53:33 +1000 Hello all, I have created Windows NT binaries for rsync 2.3.1 and have decided to make them available for others to download. These binaries have been tested on Windows NT Server 4.0 SP5 and WILL run in daemon mode if you follow my instructions below. I make no guarantees about these binaries, they have however been working for me for weeks on several NT machines. Binaries at ftp://ftp.minn.net/usr/mmchen/ Instructions for running in daemon mode: 1. You will need two files, rsync.exe and cygwin1.dll. Place rsync.exe anywhere you like (I chose c:\program files\rsync\rsync.exe) and put cygwin1.dll in c:\winnt\system32 2. You will need a program from the NT Server resource kit called srvany.exe. This program allows you to run any executable as a service. If you simply install the entire service pack it will be located in c:\ntreskit 3. Create a service for rsync by typing the following: instsrv Rsync "C:\ntreskit\srvany.exe" 4. You should now have a new service called Rsync and you can verify by looking in Start->Control Panel->Services DON'T START IT YET! 5. If you want to run rsync in daemon mode you will need a configuration file. Here is the one I use, call it rsyncd.conf and place it in the same directory as rsync (C:\Program files\rsync\rsyncd.conf) use chroot = false strict modes = false hosts allow = * [backup] path = / read only = yes list = no This example configuration will make one big anonymous anonymous rsync area available, I use this to backup my NT machines from a central Unix machine. This configuration might not be ideal for you, change to suit your tastes. The first two lines are important for rsync to work on Windows NT however. 6. You are going to need to hack some keys in the registry to make it work. Don't do this unless you are comfortable with the changes! Run regedit32 and add the following keys and values (quotation marks ARE IMPORTANT): HKEY_LOCAL_MACHINE->SYSTEM->CurrentControlSet->Services->Rsync Edit->Add Key-> Key Name: Paramaters Edit->Add Value-> Value Name: AppDirectory Value: "C:\program files\rsync" Edit->Add Value-> Value Name: Application Value: "C:\program files\rsync\rsync.exe" Edit->Add Value-> Value Name: AppParamters Value: --config="C:\program files\rsync\rsyncd.conf" --daemon 7. That's it, you should be able to start and stop the rsync service at will using the Services Control Panel. When running with the above configuration you should be able to test by attempting to telnet to port 873 from a remote machine. telnet rsync.server.com 873 (replacing rsync.server.com with your own server's address) You should get a connection to the rsync daemon running on your NT box. 8. If you have problems you are on your own, sorry, I have enough to do :) I would suggest triple-checking your spelling on EVERYTHING (filenames, configs, reg keys). If you have any comments or suggestions I would be happy to hear them at [email protected]. Mike McHenry Systems Administrator MinnNet Communications, Inc.
Delivered-To: [email protected] From: Bill Campbell [[email protected]] To: [email protected] Subject: Re: rsync: Re: using rsync to backup windows workstations Date: Wed, 14 Nov 2001 11:45:20 -0800 On Wed, Nov 14, 2001 at 10:15:59AM -0500, Thomas Lambert wrote: >I have about 12 NT servers that I want to backup 1 directory to my Linux >Server. The problem I am having is trying to get it not to ask for the >user's password. I am trying to use ssh. I just installed the latest >cygwin on my NT server (1.3.4). Rsync is 2.5.2. What I have tried was on >the NT machine, I ran ssh-keygen (no passphrase) and then added the >identity.pub to the Linux box in the /home/thomas/.ssh/authorized_keys. Instead of using ssh, why don't you use straight rsync, with rsync running in server mode on the Linux box. You should probably put separate entries in /etc/rsyncd.conf for each Windows box something like the one below that I use to backup our local hardware suffering from the Microsoft Virus, W2K. [dumbo_backup] uid = root gid = root path = /rd0/backups/dumbo read only = false use chroot = false comment = Backup for Dumbo Win2K hosts allow = 192.168.253.28 hosts deny = * Bill -- INTERNET: [email protected] Bill Campbell; Celestial Software LLC UUCP: camco!bill PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676 URL: http://www.celestial.com/ Breathe fire, slay dragons, and take chances. Failure is temporary, regret is eternal. |
Getting Rsync to work on Win NT in Services/deamon modeRefer: http://v.iki.fi/~vherva/cygwin-rsync/From: Ville Herva [[email protected]] To: Multiple recipients of list RSYNC [[email protected]] Subject: [PATCH] Cygwin tweaks and some questions X-URL: http://lists.samba.org/ X-Comment: Discussion of rsync package X-Mailer: Mutt 1.0pre3i Date: Sat, 15 Apr 2000 04:39:41 +1000 [I understand perfectly if you don't want this stuff into the mainline tree, but perhaps somebody finds it useful. Anyhow, I would be glad if somebody savvy commented especially the second point.] I just compiled rsync for CYGWIN, and found that while it compiles cleanly out of the box (newest (almost) cygwin1.dll snapshot and gcc-2.95.2), some tweaks are necessary to get most out of it. (2.4.3 was only released during my tweaking so I've been lazy enough not to upgrade. This patch is against 2.4.2). Firstly, the only sensible way to run something as "daemon" under NT is to run it as service. One can use SrvAny.exe from the NT resource kit for this. However, as it is, rsync immediately forks another process, and net stop has not desired effect, since srvany can't kill the rsync daemon. I added --dont-fork option so rsync can be forced not to fork when run as daemon. This way starting and stopping services works right. Secondly, if I connect to rsync daemon from another machine and hit ctrl-c at the client end during transfer, the rsync daemon exists (not just the connection handler process, but every rsync). I guess this is because rsync gets sigpipe signal, which is handled by sig_int, which in turn sends sigusr to the parent. I have no idea why this should be. I changed sigpipe handler to SIG_IGN, and I now get the behaviour I want: if there is a network error (such as premature socket closing) the rysnc daemon won't die, and I can reconnect to it. I didn't do this cleanly: the connection handler process should in fact exit (perhaps after some clean up), but not the parent. This is not a problem, however, since the parent will reap the connection handler after a minute. Am I missing something here, or why is sigpipe handled by sig_int? Third, under CYGWIN, it makes little sense to force attribute check on the password file. It can be made to work if env var CYGWIN contains "ntea" (nt extended attribute mode -- CYGWIN emulates unix permissions through a hack), but frankly, it is a MAJOR pain in the ass to get working especially when trying to run rsync as a NT service. Besides, CYGWIN permissions won't give you any added security -- the security should be handled with NTFS ACL's. So I added a #ifndef __CYGWIN__'s around the check. Lastly, the original reason I began compiling my own version was that I had a binary version that did not force binary open (I got corrupted .doc files etc in the transmission.) I found out that the mainline rsync source (as of 2.4.2) already has that O_BINARY bit in do_open, so no problem here. It works without, if you mount everything with -b (binary) in CYGWIN or force that with "set CYGWIN=BINMODE", but O_BINARY is the (most) correct way to do it -- it works regardless how the user has mounted his filesystems. However, if the mounts are binary, the configuration files are not read right: if, for example, the user makes a password file with notepad, he'll get an additional CR into the password (due to missing CR/LF->LF conversion). So I added open(O_TEXT) forcing to every config file open (#ifdef CYGWIN'ed, of course). Now the users shouldn't have to worry about mount types. I put the patch at [http://v.iki.fi/~vherva/cygwin-rsync/]. [OT] In case you are wondering: I'm implementing NT workstation backups with rsync here. One Linux server with two 34GB ide disk softraided together and e2compr patch thrown on top of that. Then I just take incremental backups with rsync so that every file is a hardlink to yesterday's copy unless it's changed. (Basicly, cp -al yesterday today;rsync rsync://nt/dir today/) Add e2compr to that, and the backup won't really take that much space anymore. Just the directory structure and changed files (compressed) per backup. The nice thing is that I can export the whole beast with samba, and the users can access their yesterdays or monday's tree transparently without hassle. No need to disturb operator and to showel tapes just to find out that, gee, we have no backup of THAT file. Of course, there are certain gotchas, but overall, it seems to work quite nicely. |
Getting Rsync working happily with Secure Shell for UNIXAlso refer: Rsync/ssh backup scripts: ftp://ftp.stearns.org/pub/rsync-backup/
From: "Adye, TJ (Tim) " [[email protected]] To: "'John Horne'" [[email protected]] Cc: "'[email protected]'" [[email protected]] Subject: RE: SSH with non-default key Date: Fri, 1 Dec 2000 14:13:33 -0000 Hi John, > I am trying to work out how I can use rsync with ssh but also > specifying the > key to use. I want to run rsync/ssh via cron but need to use > different keys > from the same account. With just ssh this is not a problem - > the -o option takes care of it. Which version of ssh is this? I use Unix ssh 1.2.x, and the option is -i. I run rsync with rsync -e"ssh -x -a -i IdFile" (-x -a turns off unnecessary X11 and agent forwarding for speed in setting up). If you want to specify the identity file as an option with -o, you need to use -o IdentityFile=IdFile (or if you really like to play with quotes, -o 'IdentityFile IdFile'). Actually I use this so often that I have a wrapper script that allows the -i option (and several others) to be specified directly and then builds its own ssh command. > However, I can find no option to rsync except perhaps by > using '-e ssh'. Changing this to something like '-e "ssh > -o'IdentityFile2 > ~/.ssh/newkey'"' doesn't work (yes, you need to escape some > of the quotes). What's this IdentityFile2 option? That's a new one to me. > Setting the environment variable didn't seem to work either > (I may have got > things wrong, but it just didn't like it). It seems that > rsync keeps seing > the -o option (or rather the key I specify) and complaining > about it. Tim. ============================== cut here ============================== Tim Adye, BaBar/DELPHI Groups, Particle Physics Dept., _ /| Rutherford Appleton Laboratory, UK. \'o.O' Oop! e-mail: [email protected] or VXCERN::ADYE (HEPNET/SPAN) =(___)= Ack! WWW: http://hepwww.rl.ac.uk/Delphi/Adye/homepage.html U Thphft!
From: John Horne [[email protected]] To: "[email protected]" [[email protected]] Subject: RE: SSH with non-default key Date: Fri, 01 Dec 2000 16:01:58 -0000 (GMT) On 01-Dec-00 at 14:59:38 John Horne wrote: > On 01-Dec-00 at 14:13:33 Adye, TJ (Tim) wrote: >>> I am trying to work out how I can use rsync with ssh but also >>> specifying the key to use. I want to run rsync/ssh via cron but need to >>> use different keys from the same account. With just ssh this is not a >>> problem - the -o option takes care of it. >> >> Which version of ssh is this? I use Unix ssh 1.2.x, and the option is -i. >> I run rsync with >> >> rsync -e"ssh -x -a -i IdFile" >> Okay I've got this fixed now :-) I've been trying to use something like: -e "ssh -o'IdentityFile2 ~/.ssh/xxkey'" for some reason something doesn't like the arrangement of quotes (I have tried swapping them around but to no effect). I don't know if its rsync, ssh or the Korn shell I'm using (on a Sun system). Anyway, by using: -e "ssh -oIdentityFile2=~/.ssh/xxkey" this works fine :-) The first form is valid, but as said something doesn't like it. Thanks for your help, John. PS - for those reading this and use openSSH from cron and kept getting the 'Pseudo-tty will not be allocated...' message back (very annoying), I have received a patch for this since I pointed out on the developers list that the '-T' option was supposed to prevent a tty being allocated (and hence the message was not necessary). Hopefully the next release of openSSH will have this fixed once and for all :-) ------------------------------------------------------------------------ John Horne, University of Plymouth, UK Tel: +44 (0)1752 233914 E-mail: [email protected] PGP key available from public key servers
From: Dave Dykstra [[email protected]] To: Ernie [[email protected]] Cc: [email protected] Subject: Re: strange rsync problem Date: Fri, 1 Dec 2000 15:52:47 -0600 On Fri, Dec 01, 2000 at 01:41:53PM -0500, Ernie wrote: > > You can specify an explicit path to rsync on the client side with --rsync-path. > > Oh, duh, i should have remembered that. I did use the --rsync-path paramter > with no success. I presume by "no success" you mean that it still hung, right? I would think it should at least have had success finding the remote rsync. The rsync-path parameter is really a shell command for whatever login shell you're using on the remote side so you may have some success debugging it by using something like --rsync-path "set -x; /usr/local/bin/rsync" or --rsync-path "strace -o /tmp/rsync.strace /usr/local/bin/rsync" You can also turn on ssh debugging with -e "ssh -v" > Hm. I am running rsync 2.4.6, with OpenSSH 2.3.0p1. Both boxes have this > config, and are running linux: - Dave Dykstra |
Why Rsync may not be copying files overFrom: David Bolen [[email protected]] To: "'John Horne'" [[email protected]] Cc: [email protected] Subject: RE: Changed file not copied Date: Fri, 1 Dec 2000 13:03:38 -0500 John Horne [[email protected]] writes: > Okay I think I've solved this :-) The rsync with the '--size-only' > option updates the modification time for the file but doesn't copy > it. Actually, I believe it's less related to "--size-only" rather than your use of the "-t/--times" option (implicitly since you used "-a"). Thus, "--size-only" says not to bother transferring the file contents even if the date is different but the size is the same, but the "-t" still transfers over the appropriate timestamp information. This can be a good way to initially sync up two systems that weren't previously mirrored with maintenance of timestamp information, but otherwise have the same content. > Hence when I run the command without that option the size and time > are equal - hence the file is not copied despite being different. I > see that there are options to ignore the time as well, so that would > get round it on the second rsync command. Not a problem, I just > found it a bit confusing :-) In addition to ignoring the timestamp ("-I/--ignore-times"), and depending on your environment and size of the files involved, you can also use the "-c/--checksum" option to have rsync compute a file checksum to determine if a file should be transferred. While this can be slow, if the actual rsync processing of the file (e.g., block checksums) and transmission of that information (particularly on a slow link) would be lengthy, it's a more efficient way to guarantee you only bother sending a file if it is different. -- David /-----------------------------------------------------------------------\ \ David Bolen \ E-mail: [email protected] / | FitLinxx, Inc. \ Phone: (203) 708-5192 | / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \-----------------------------------------------------------------------/ |