How to mirror a backup?

general discussion
1

I wonder how to do mirroring of backups and things. I am using a partition formatted with btrfs as a backup destination, on a disk connected to a raspberry PI. Now I would like to have a copy of it (to another disk), what I store in another place, for safety, so it would be not a permanent mirror, just connecting it once a week to copy the new diffs. What do you recommend for this? I was thinking of a RAID1, using btrfs’s raiding features, but then I dropped the idea: it would be quite tiresome to regularly downgrade and upgrade the raid, and I am not even sure that it can be done with only 2 disks, where only 1 would be present most of the time. Then I thought of making a ‘backup of the backup’, but I don’t know if I can do that with urbackup: there is one target directory, as I see it, and you cannot backup it to another place (with the same server). Perhaps I should use a ‘traditional’ Unix program, like rsync. Is there some btrfs feature which can help me here? Maybe the snapshot?

Thanks,
Zsolt

1 Like
2

Btrfs has (like ZFS) the really cool send/receive feature, where you can send filesystems and differences between filesystem snapshots to another filesystem. See https://btrfs.wiki.kernel.org/index.php/Incremental_Backup . For ZFS this is in the admin manual: http://urbackup.org/administration_manual.html#x1-680009.7.1

1 Like
3

Hi, a couple of thoughts…

  1. Firstly using ZFS it is possible with snapshots. I have been able to successfully mirror to another server using the info from the Admin manual. Snapshots are able to only transfer the differences but there is quiet a bit on load on both sending and recieving servers when this happens. Details of this are in the Admin Guide http://urbackup.sourceforge.net/administration_manual.html#x1-680009.7.1

  2. Rsync. Rsync is my prefered method of transfer as it is filesystem independent. You can transfer from 1 type of filesystem to another (ie ZFS ->btrf or ext4). Only differences are actually transfered but both sending and recieving servers have to do quiet a bit of work to figure out the differences.

Here is an example of my rsync command for transfering…

rsync -avzHh --delete --exclude=urbackup_tmp* -e ssh /mnt/backups/ username@moon.someserver.net:/mnt/mirrorcopies

The H rsync option is used to preserve hard links, z to compress transfers, and a is archive mode with preserves everything basically except hardlinks but we have already specified this with the -H option. --delete option tells rsync to remove any files on the recieving server if they are not present on the source server. --exclude tells rsync to ignore urbackup temporary files.

The command is using ssh to connect to the remote server securly (a password will be required or a preshared key for ssh to connect)

“/mnt/backups/” is my live backup mount and “username@moon.someserver.net:/mnt/mirrorcopies” is the remote location (ie: /mnt/mirrorcopies on server moon.someserver.net)

Hope this might be of some help.

Regards,
Brad.

4 Likes
4

I am using rsync with -aH to mirror a backup to a different server.
But what I get are a lot of “rsyncd-munged” symlinks pointing to the original backup server. How can I set up rsync to create symlinks pointing to the mirror server?
Or should I resolve the symlinks by using -L option?
The backup server has ext4 FS, mirror server has btrfs.

1 Like
5

The “rsyncd-munged” symlinks have been caused by “Use symlinks during incremental backup” being enabled.

6

Do you recommend unchecking “Use symlinks during incremental file backups” option then?

7

I would say it depends on your use case.
I have now disabled the option. Backups on clients are still running fast enough.

8

For backing up the whole btrfs volume to another server (other may be would use an external usb drive) we use btrk, seams to play nice together with urbackup.

  • we haven’t found an option to use btrfs send/receive the whole btrfs partition including all snapshots to other server/drive, yet

  • so, for now I use btrbk to pull (!!) all “current” btrfs snapshots of all clients to other server

  • if you just use an external drive, you could use “btrbk archive” feature (doesn’t work over ssh)

Here is how we do that:

  • install btrbk

  • setup openssh public key config to urbackup server using http://digint.ch/btrbk/doc/ssh_filter_btrbk.html

    from="(your external server ip)",command="/backup/ssh_filter_btrbk.sh -l --source --delete"

  • have a cron job:

    for i in /etc/btrbk/btrbk.conf.* ; do /usr/sbin/btrbk -v -c $i run ; done

  • have a /etc/btrbk/btrbk.conf per client:

      # btrbk config for client XXX
  # change where XXX
  #
  transaction_log            /var/log/btrbk.XXX.log
  snapshot_create            ondemand
  target_preserve             4m 
  
  ssh_identity               /etc/btrbk/ssh/id_rsa
  ssh_cipher_spec            arcfour
  
  volume ssh://server.domain.tld/srv/clientbackup/
    subvolume     urbackup/XXX/current
    snapshot_dir  urbackup
    snapshot_name XXX
    snapshot_preserve_min latest
      target send-receive      /srv/yourBTRFSdestination/server.domain.tld/urbackup/

This is not a usable urbackup volume then, but the data is there and we can use btrfs snapshots and btrfs send/receive.

The difference is: urbackup uses snapshot names urbackup/clientname/1707dd-hhmm while btrbk uses urbackup/clientname.201707dd

May be someday btrbk can support other name schemes, too, but for now we have a recent snapshot from every client. For windows image snapshots you could adjust the btrbk.conf.xx files accordingly, perhaps with some scripting.

2 Likes