How to test a disaster recovery for a Linux server while the original server is still running

We use urbackup to completely backup a linux based server.
Now we would like to test the disaster recovery where we’d need to restore everything on bare metal.
Is there anyway to do that, without interrupting the still running server and especially without stopping the urbackup client there? It should continue backupping it’s data!
From other threads I’ve read what I found out how to do a bare metal restore would be

  • Booting up a linux
  • install urbackup client
  • get the authentication details of the previous urbackup client for which you want to do the restore
  • restore to filesystem, databases, etc.

I see the problem with using the same authentication details. I’m not sure how that would affect the ongoing backups from the original server.

  • How does the urbackup server react when do clients with the same authentication try to connect?
  • Would it at once try to start backups from the wrong client (the one we try to restore?).
  • Would the old client still be able to do backups?

Would it be somehow possible to create a new client which can access the backups of an other one?
It is possible to create a urbackup user which is able to access those backups, but I don’t really see, how I could use that from a shell…

If somebody could shed some light on the issue, i’d be thankful :-).

The “urbackuprestoreclient” running on the restore CD would be the place to add this, I guess. There is actually code in this client and on the server to restore files after username+password login (like with image restores). It just (currently) isn’t exposed via command line and not tested (so it probably won’t work :wink: ).

It would be great if that would be implemented.
If you backup important data, it’s good if from time to time you can test if you can restore all your data at once, without interrupting the currently running server and it’s backup processes.

But as far as I’ve meanwhile. It seems to be incredibly difficult to restore the data (so I’m really happy I’ve tested that out, and in the emergency case, I’ll now how to do it, because it’s not obvious at all).

So first the naive approach which doesn’t work:

  • stopped urbackupclientbackup service on the original server

  • I’ve download from the web-interface “download client for linux” the urbackupclient for the server I want to restore.

  • installed the downloaded client on a clean server and connected it

  • The connection was successfull. But that’s about the only thing that is successful…

  • It’s not possible to browse old backups, you only get an error message saying that there was an error. even using debug loging on urbackupclientbackend doesn’t reveal anything more.

  • Since I’ve read it in an other thread (several years old), I’ve tried first to do a backup of some random data. Afterwards the browse command worked, but ONLY showed the data I’ve just backup, none of the old ones.

  • I’ve managed to get the backup-id of the old backups from the web gui.

  • but even then, when trying to do a restore-start, nothing happens. when using the command, which is supposed to be the right one:

urbackupclientctl restore-start -b 1827 -d root -m / -t /root/test

(It took me quite a while to get those parameters. The documentation doesn’t explain well enough how to use them) The Result is:
Error starting restore. Errorcode: 4
When trying out (what should be incorrect, but was the first way I’ve tried it out)

urbackupclientctl restore-start -b 1827 -d / -m / -t /root/test

The result is:
Restore completed successfully.
But 0 bytes copied.
And in any case 0 infos in the debug log.

But I don’t give up that easily. So

  • On the new client I’ve deleted /usr/local/var/urbackup/*
  • Copied the whole /usr/local/var/urbackup/* directory from the original server
  • started up urbackupclientbackend
  • Now the browse command works, but o wonder, doesn’t shows the “test” backup I’ve just done before, but all the old backups.
  • and unbelievable but true:

urbackupclientctl restore-start -b 1827 -d root -m / -t /mnt/restore

Now works!!!
So in about 20 hours I will see if at the end of the process I’ll have a working server again. Unfortunately, the real server can’t do any backups meanwhile :-(.

So it seems it is possible to restore a linux server on barebone. But man, it’s not straight forward. And not really well documented (e.g. that you need to copy the previous var/urbackup directory instead of just downloading a new client from the gui). Also the error messages urbackupclientctl give are basically useless. They just tell you there was an error , and that’s it (or even tell the command was successfull eventhough it wasn’t).

btw. If anybody want’s to know it: On the new server I didn’t install a new linux system, I’ve just started up from a grml cd (live linux for administrarion purposes), and there directly installed the urbackupclient (it will be installed in the ramdisk but works perfectly). From that live linux you can also recreate your partition layout, logical volumes, mbr, etc.

Thanks for the report…

The method with downloading a client specific client from the server should work. If you want to investigate further why this doesn’t work…

The installer has data embedded with access keys. If those aren’t present you won’t be able to access/restore the backups. So first make sure the user you are downloading the client with can browse the backups of the client (in the web interface).
Then you could check if the downloaded client actually contains the keys. Search for 48692563-17e4-4ccb-a078-f14372fdbe20 in the client installer and inspect the access_keys line. The line should have a uroot:<ROOT access token> and t<server token>=<client access key>. After installation and startup they land in /usr/local/var/urbackup/ and /usr/local/var/urbackup/tokens/user_726f6f74. Since only the root key is present, you should browse/start the restore with root as well, otherwise backups won’t show up. But for newer versions you should run full system restores as root anyway, because other users might not be able to access some files, so they wouldn’t restore in that case.

First, the good news. The server was running perfectly after the restore :-).

  • urbackupclientctl and urbackupclientbackend are both run as root user
  • The user in the web ui for downloading the client is the admin user, so it should have access to everything.
  • I’ve found the requestet line, the whole thing looks like that:

#Initial Settings. Changes will not be respected.
internet_server={server - name}
computername=Intranet Hebammen 2
access_keys=t{some letters and numbers}:E{some letters and numbers}

for easier access lets give it some names to the key parts, so let’s call the parts which we see above:
access_keys=t{key 1}:E{key 2}

I’ve also verified the other two files on the original client and post them here, since I’ve got absolutely no idea how to read them
/usr/local/var/urbackup/ and /usr/local/var/urbackup/ and /usr/local/var/urbackup/tokens/user_726f6f74 look the same:

key.={new key 3}
key_age.{key 1}=1566999918
last.key.={new key 4}
last.key.{key 1}={new key 5}
key.{key 1}={key 2}
key_age.{key 1}=1567653118

There was a bug I’ve noticed during the restore process.
The root filesystem “/” is on a logical volume and during the backup process a snapshot is create of it.
The boot filesystem “/boot” is directly in a partition “/dev/vda1” so obviously no snapshot can be created of it. The directory to backup looks like that in urbackupclientctl:

/ root follow_symlinks,symlinks_optional,share_hashes

So it should be allowed to also backup other filesystems. But it doesn’t. “/boot” is completely empty in the backup. And there is no Error or Warning or Info about it in the backup log.
I’ve tried out to do the backup without creating a snapshot of the root filesystem, and then it also backups the “/boot” directory. Fortunately the /boot directory is relatively easy to restore manually but obviously that could also happen with some need-to-have data.

There is an other problem. I’m not 100% sure if it is related. After I’ve activated the urbackupclientbakend on the original server again, it can’t to backups anymore.
What happened:

  • urbackup started doing one successful backup from my client which was restored, so from the whole live linux system and everything.
  • I’ve disabled the urbackupclientbackend on the restored server
  • activated urbackupclientbackend again on the original server
  • What happens now, each time it tried to do a backup following error appears. It still continues copying data for a few hours more, but at the end the backup fails and is not stored. I’ve attached the log as a file: urbackup.log (2.9 KB)

As I’m saying. I’m really not sure if it’s related. But not a single Backup from the original server worked since I’ve switched the clients. I’ll try to start the clean up process and see if it will change anything.

Thanks for your time :-).

Ok that one I could solve, by doing a new Full File Backup instead of only the incremental one.

The random data you backed up on the restore client… is it e.g. /etc of a distribution or something, in case I want to reproduce the last problem… (and what was the distribution of the backed up Linux)
I’m guessing you are using directory symlinks, correct?

  • The distribution of the backed up Linux is Debian Stretch.
  • THe backuped up directory has following attributes: follow_symlinks,symlinks_optional,share_hashes

What happened exactly was:

  • Installed the downloaded client from the web ui which could not see the backups, and added the directoy “/root” to do a backup of it. There were maybe 4 files in there
  • Copied the client settings ("/usr/local/var/urbackup/") from the original server and did the restore.
  • At one point, propably after the restore finished urbackup started the backup job on the restored server, so it did a complete backup of “/”. Including the live Linux and everything, since that was in the settings which I’ve copied from the original server. The live Linux was GRML
  • After it’s already done one successful backup, I’ve noticed that it was doing backups of the restored server, so I’ve aborted the backup and stopped the urbackup client backend on the restored server
  • One day later once the backup job finally disappeared from urbackup, I’ve started up the urbackup backend on the original server. And the above error appeared, until I’ve did a Full file backup.

Hopefully I’ve wrote that down in an understandable and reproducible way :-).