Way to duplicate the file store?

general discussion
1

I think urbackup is awesome. But besides having a good local backup, we are wanting to make a duplicate of the backup drive to USB (or something) that we can carry offsite. We are using the Windows urbackup server 1.3.2. I have a nifty copy of rsync that handles hard-links, but it takes an incredibly long time to do a backup. Does anyone know of a way to create a duplicate of the urbackup destination space for hauling offsite?

2

You can try robocopy, considering it’s on a newer server. I’ve noticed that Robocopy can be quite speedy sometimes, using the right settings. Also depends if it’s going to a NAS or USB drive. And if the USB is 2.0 or 3.0.

3

From my understanding of things, you cannot copy hard-links (and keep them hard-links) to a drive that is on the network. Hard links need to be written to something that is locally attached. I may see if I can add an esata port to the server to which it is attached; the USB2 port is pretty slow. But just watching how the copy goes is a little painful. I have been duplicating the drive for over a week now, and it still has a ways to go. But I have to admit, it is pretty impressive to see 12TB of individual files on a 2TB drive, and still have some room. I really like the file de-duplication; it just increases the complexity of duplicating the data store.

4

rsync can be configured to copy hardlinks to a network share and once the bulk of the data is there repeats of the sync will only transfer changed data, generally making subsequent backups much quicker. Unfortunately when hard-links are involved there is alot of work for rsync (on both machines) to figure out what has changed. I not sure if there are any catch 22’s for running rsync on windows with NTFS.

I have had a lot of success using rsync on windows (via cygwin) but I have not had a need to copy hardlinks. I can try some tests if you like.

5

rsync can be configured to copy hardlinks to a network share and once the bulk of the data is there repeats of the sync will only transfer changed data, generally making subsequent backups much quicker. Unfortunately when hard-links are involved there is alot of work for rsync (on both machines) to figure out what has changed. I not sure if there are any catch 22’s for running rsync on windows with NTFS.

I have had a lot of success using rsync on windows (via cygwin) but I have not had a need to copy hardlinks. I can try some tests if you like.[/quote]

What program do you use for that? Personally I have seen some luck using Delta Copy. I use that to make on site backups for some of my clients, sends the files right to a NAS.

6

I use Cygwin ( http://www.cygwin.com/ )

my current version of rsync (via cygwin) provides following rsync features …

$ rsync
rsync  version 3.0.9  protocol version 30
Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    no socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, no xattrs, iconv, symtimes

A couple of examples…

Destination filesystem mounted on same machine as source…

rsync -avzHh  /mnt/backups/  /mnt/backup_copy/

Destination filesystem mounted on another host (rsync & ssh need to be available on both servers - can use cygwin for this as well)

rsync -avzHh  -e ssh /mnt/backups/ urbackup@other.server.net:/mnt/backup_copy
7

Thanks for the examples.

What bjharper both said and showed, was that one can do it with an rsync agent running on both computers (source and destination).

I believe that, for both robocopy and rsync, you cannot connect to a remote machine (that does not have an agent on it) and copy hard-links across CIFS (Microsoft Windows File Sharing). The drive either needs to be local, or there needs to be an agent (rsync or something similar) that is local to the drive that you are writing to. I am pretty sure you cannot, for example, mount a Z drive to a NAS and use a “local” copy of rsync to copy to the Z: drive.

So, if the NAS has enough of an OS to run rsync, then you can duplicate a URBackup drive to it. But if it only allows you to connect via a Windows Share, you are out of luck.

I hope to get to the point where I can try this on an eSata drive to see if the performance becomes reasonable. I do not think my USB2 connection will even finish syncing the first copy over. After a week it had about a third of the data copied. (sigh)

8

Yes this might be one of those catch 22’s, I don’t have that much experience with windows file sharing. If I can find some time ill try in the next
few days a test for you.

9

I could not sleep had to test, results success :)

brad@brad-PC /cygdrive/c/test
$ dir -l
total 3
-rw-r--r--+ 3 brad None 11 May  8 00:36 1
-rw-r--r--+ 3 brad None 11 May  8 00:36 2
-rw-r--r--+ 3 brad None 11 May  8 00:36 3  

brad@brad-PC /cygdrive/c/test  

$ rsync -avzHh * /cygdrive/z/test
sending incremental file list
3
2 => 3
1 => 3  

sent 118 bytes  received 49 bytes  334.00 bytes/sec
total size is 33  speedup is 0.20

/cygdrive/c/test is my local machines disk (NTFS)
/cygdrive/z/test is a remote NTFS share mapped as Z: drive
files 1,2 & 3 are all hardlinked together (and confirmed hardlinked after rsync on remote server)

Both machines Windows 7

10

You probably don’t want to do it on a per-file basis, but on block level.

For example you could put the two drives in a software raid-1 and sync them. (Windows has a build in software raid capability)

I’m sure there is a free app that can make a shadow copy of a disk and copy it on a per-block basis to another one. A few seconds of googling got me DriveImageXML. If you come up with a better one please post it! (All the code to clone a drive is already in UrBackup btw. its just writing to another drive vs. over network into a vhd)

11

I had wondered a little about that. I was thinking that it would have had to have a fair bit of that code, though a lot depends on how it was written. It probably would not be all that easy to build in the ability to clone the data store (or, in particular, to have a clone of the data store that you can “update” without needing to duplicate the whole thing every time)

Thanks for the thoughts on the “per block” copy. That makes sense. It certainly could have done a copy much faster than rsync has been doing it…

12

[quote=“uroni”]
I’m sure there is a free app that can make a shadow copy of a disk and copy it on a per-block basis to another one. A few seconds of googling got me DriveImageXML. If you come up with a better one please post it! (All the code to clone a drive is already in UrBackup btw. its just writing to another drive vs. over network into a vhd)[/quote]

Having first-hand experience with DriveImageXML I don’t fully recommend it. It can be quite clunky. Restoring is decent but weird to complete. I recommend Drive Snapshot:

13

I use Cygwin ( http://www.cygwin.com/ )

[/quote]

OK then, Delta Copy uses Cygwin as well. The nice thing about Delta is it runs a service (Windows boxes) for the server and a client can be run on the same machine. It is also very easy to schedule it and set it up. It’s more than likely an easier program for the GUI-minded folks out there.