Bare metal encrypted disk restore

First let me say that I already have a working solution, although it isn’t optimal as it must be done manually and offline. I’m hoping that UrBackup will provide a better solution.

My backup requirements are somewhat unusual.

The system in question comprises a 1TB HDD with a primary, hidden and active NTFS formatted 105MB “System Reserved” boot/recovery partition, and the remainder as drive C - a primary, NTFS formatted system partition. The C partition is fully encrypted with Veracrypt, and requires a password at boot to activate.

To be more precise, the disk layout begins with the Veracrypt bootloader, the MBR, the first partition VBR and an NTFS filesystem (System Reserved), followed by a second copy of that partition’s VBR.

What would normally follow would be the second partition VBR, but in this case is just encrypted data. Only the first 512 bytes of that encrypted data are needed for Veracrypt to mount the encrypted partition (but not the filesystem), and map the decrypted version to a virtual device. If a valid filesystem is present on that virtual device then it can be mounted. If not then it can be created.

Under Linux, this Veracrypt virtual device is presented by device mapper as /dev/mapper/veracrypt[n], where “n” is incremented for each Veracrypt device that is mounted (although it’s more technically correct to say “activated”, since no actual filesystem is mounted yet). To Linux, this looks like a regular partition (but, importantly, not like a whole disk).

Similarly under Windows, this Veracrypt virtual device is presented as e.g. \?\Volume{60b38509-f955-11d8-bdab-806d6172696f}\ (using GUIDs) or \device\HarddiskVolume4 (using paths), and then assigned a friendly drive letter, e.g. C. The underlying real partition is seen by the system as a raw, unformatted device, and is not assigned any drive letter.

Every backup solution I’ve tried thus far, both open source and commercial, has failed to correctly restore this system to a pristine working state. The only solution that works is one I devised myself using a combination of other tools. However it isn’t automated, it must be done offline, and I suspect that it results in much larger backups than necessary.

For example, Acronis True Image, for all its sophistication, is quite incapable of restoring this system to a bootable state, even using its “Universal Restore” (to different hardware) feature, and even using Microsoft’s own recovery tools afterwards (bootrec, bcdedit and friends) has no effect.

The problem is that whole disk/partition encryption is necessarily invisible to the OS, and therefore any backup software running within the live Windows environment is incapable of recognising that the partition it’s imaging is actually virtual, and that the characteristics of the real hardware underneath are very different.

Worse still, disk imaging software like True Image gets very confused by such virtual devices, and misinterprets them (and then backs them up) as something quite unusable for restore purposes, so even though the backup process appears to work, the restore will unexpectedly fail, with no possibility of recovery.

To give you an idea of exactly how confused Acronis software gets, I once constructed a rescue meda that combined Acronis Disk Director and True Image with Veracrypt, so that True Image would be able to see the decrypted filesystem. The result was that Disk Director saw the decrypted partition as a “super floppy”, and all of its various partition modification tools were greyed out. True Image refused to display or work with the decrypted filesystem at all.

This is why it’s so important to test your backups … actually test them with a full restore, not merely verify the backup’s checksums.

I find it very disappointing, and somewhat perplexing, that the backup industry has still not developed any strategy for bare metal restore of encrypted storage, despite the exponential growth in public interest in disk encryption in recent years, culminating in Microsoft and hardware vendors building support for it right into the BIOS and OS. This is no longer a corner case, and needs to be more widely supported. Backup should not be viewed as “optional”.

My solution is rather convoluted, but works, and must be run offline using a Linux system (in my case on a USB thumbdrive).

It begins by having the foresight to make a disk image dump (using dd) of the bootsector, MBR and everything up to and including the first 512 (encrypted) bytes of the second partition. This is only 105MB, but is crucial.

(I should also note that the obvious answer of simply using dd to image the whole, raw, encrypted drive doesn’t work, since it end up interpreting some of the random data as bad blocks, and the restored dd is then unbootable. Using ddrescue instead may help, but even then I’d end up with a massive file containing both used and unused sectors, with no easy way to do incrementals, deduplication, or mount individual filesystems within the image.)

I then “mount” (activate) the encrypted partition using Veracrypt, and run ntfsclone on /dev/mapper/veracrypt1, piping the output to a deduplication backing store called “borg”.

Recovery is simply the same process in reverse.

This gives me pristine copies of only the filesystem, which I can then restore manually using ntfsclone, after recreating the partition layout using dd, and activating the encrypted partition with Veracrypt.

However, I dislike having to reboot just to make backups, and I’m not convinced that borg deduplication is as efficient as VHD differentials, either from the perspective of time or data size, especially if the resulting differential VHDs are also combined with a deduplication system.

So, here are my questions, with respect to using UrBackup:

  1. If I am always backing up a Windows system partition under Windows, and always restoring it offline using a Linux system, do I actually need to make image backups at all, or can I restore a pristine copy of the NTFS filesystem only, without any partition layout data, using UrBackup under Linux, only from a normal file backup? In other words, can Urbackup make a full and pristine backup of a Windows system partition filesystem using only a file-level backup method in the first place, and if so then does the subsequent restore process under Linux properly handle all properties of this NTFS filesystem, including ADS, junction points, standard file attributes and ACLs, to produce a filesystem that is a pristine copy of the original?

  2. Can I have a brief explanation of the deduplication methods employed by UrBackup, and how this compares with bup, fsarchiver, borg, btrfs, ZFS and other deduplication systems? How does the deduplication of image backups compare with deduplicated file backups, in terms of speed and data size?

  3. I’ve just completed a UrBackup image backup of my Windows system partition, and one incremental (synthetic) image backup, with the following stats:
    Full image backup, Volume: C:, 501 min, 404.38 GB Incremental image backup, Volume: C:, 184 min, 580.58 MB.
    How much faster would an incremental backup be using change block tracking? In order to use this feature, do I need to create a new full image backup, after installing the CBT client, or can it start with an existing backup?

  4. If I stick with image backups, in order to avoid writing invalid partition information (vis-à-vis Acronis True Image and Veracrypt virtual devices), using the UrBackup rescue media and a disk image backup, can Urbackup restore the filesystem only, without making any attempt to recreate partitions, or do I have to mount the VHDz and use something like ntfsclone to make a pristine restore? In that regard, how does one “mount” a VHDz in such a way that it exposes a device node only, rather than a live mounted filesystem, since ntfsclone refuses to operate on live filesystems? Can I do this with qemu-nbd, and how is this complicated by gzip compression and incremental image assembly? Would I need to completely reassemble and decompress the backup to a new file in order to “mount” it, or is the output file merely a handle through which data is piped?

  5. Not that I have any articular objection to the UrBackup rescue media, but I prefer using my own, full Linux system to do restores and other maintenance, not least of which because I need an installed copy of Veracrypt. Is there something unique about the UrBackup rescue media that cannot be replicated on my own Linux environment, or can I just use my own system and install UrBackup server and client? Is there any difficultly in then connecting that server to the backup made under Windows, assuming it has access to it?

My instinct tells me that I should just stick to file level backups. I just need to know if a bare metal restore is possible in that case.

Any suggestions or comments appreciated.

Ewww that was long to read

  1. Just try it.
    As i undertsand, that should work, Once the partition is unlocked, it s just files, like any parition

Bup, borg, btrfs, ZFS works by block level dedup , so you d get smaller file size
as i remember u can t do 2 backups at the same time with borg.
Peoples say what they want , but at the office we still have problems with btrfs.
zfs dedup is super ressource hungry, if you realise it s too ressource hungry and want to deactivate it , you wont have enought ressource to deactivate it.
Anway btrfs and zfs are storage mediums, not backup software. They aren t targeted to windows which is your use case.

fsarchiver, i dunno how it works

For the average use case File level dedup is a good compromise here for the backup speed, ressource usage and resulting size. If you browse the files on urbackup server, the hierarchy is clear and self explicit.

3
in theory it should be a lot faster, there s no comparison to do
else no idea

4
i am sorry i dont understand. I think your are going for something too complex.
Maybe if you have some devs skills you can suggest a patch to get the functionality you want

5
idem , maybe suggest some patch. it s usually better to maintain stuff that way

6
i think too.
in your case image backups would break too easily. so do at least files backup plus maybe try to submit patches to uroni to handle the stuff you want. When that stuff is rock hard solid, then you ll rethink your backup strategy

1 Like

I use TrueCrypt and now VeraCrypt on one of my laptops. When I restored a image backup (without testing it first :wink: ) it worked out okay. UrBackup backs up the image unencrypted, it is then restored unencrypted, the TrueCrypt bootloader allows one to skip the encryption during boot, after that I encrypted the drive again.

I am using whole disk encryption. That might make a difference since it seems you have only the C partition encrypted…

It backs up all the NTFS special stuff, but is only able to restore it on Windows currently.

It would make the incremental backup faster. Without CBT it has to read 404 GB on the client to find out that only 580MB changed.

You can restore via command line using the VHD files: https://urbackup.atlassian.net/wiki/display/US/How+to+restore+via+command+line
The VHD mounting uses the fuse file system -> loop device trick, so the volumes gets exposed as raw file and loop divice which you could ntfsclone.

See above link. You can also build the restore client from github with ./update_restore.sh (or by looking at the contents and setting the configure options).

BMR with files is probably possible, but fiddly. And you’d have to use Windows PE. No one has tried it with UrBackup yet. Here are some instructions for another backup program that should give some hints: http://burp.grke.org/docs/baremetal-windows.html

1 Like

Thanks for your responses.

Right, so it seems I really do need to use image backups.

I’ve previously tried the recommended method for restoring backups of VeraCrypt partitions, using other software such as True Image, which is as you suggest just to restore the decrypted partition then skip the bootloader authentication, then re-encrypt after restore.

Unfortunately that failed in my case (“bootmgr is corrupt”), and all attempts to recover from this error also failed. The written filesystem itself was valid, since I could access and even fsck it under Linux, but the Windows boot process was so misconfigured that I was unable to fix it. Presumably there is a way, but utilising every tool and method at my disposal failed to correct the problem. It turns out that the Windows BCD is far more fragile than other boot managers like GRUB, and its proprietary nature makes it almost impossible to work with in anything but simple, default and supported configurations, unless you’re a Windows developer with access to the APIs (which I am not).

My best guess is that bootmgr was convinced that the partition ID it needed was actually the raw, encrypted device, rather than the decrypted virtual device exposed by VeraCrypt, and nothing could convince it otherwise (bcdedit failed to find it, and I know of no other way of altering the BCD).

I think you’re right that having a “System Reserved” partition complicated matters, but this is the default layout produced by the Windows installer (there is a hack to avoid it, that I was unaware of at the time).

Unfortunately this means that Urbackup will probably fail to restore correctly for the same reason as True Image or any other imaging software, unless I can first expose the decrypted virtual device with VeraCrypt, and restore to that instead of the raw disk.

I’ll install UrBackup on my Linux thumbdrive and test this procedure, then let you know the results. I’ll test a simple restore first, without VeraCrypt, to see if UrBackup handles this situation better than True Image.

./urbackuprestoreclient --restore-image /mnt/win3/urbackup/Rocky/161124-0908_Image_C/Image_C_161124-0908.vhdz -o /dev/sda2
2016-11-24 23:39:33: VHDCopy.
2016-11-24 23:39:33: VHD-Parent: "/mnt/win3/urbackup/Rocky/161123-1933_Image_C/Image_C_161123-1933.vhdz"
2016-11-24 23:39:34: VHD Info: Size: 931.415 GB 953768 MB
2016-11-24 23:39:34: Skipping 524288 bytes
2016-11-24 23:41:09: 1%
2016-11-24 23:43:57: 2%
2016-11-24 23:48:13: 3%

Why “Skipping 524288 bytes”?

Well, the good news is it worked.

Nearly.

UrBackup got nearly everything right … except for the bootloader. VeraCrypt complained with “Disk error. Loader damaged! Repair with Rescue Disk”.

Looking at the disk with a hex editor, it seems UrBackup only restored the first 512 bytes of the bootloader, but on my disk the boot code extends beyond the first 512 bytes, and the first partition doesn’t start until offset 2047.

Fortunately I had a copy of the original bootloader, so I just restored it with dd and rebooted.

One step closer…