First let me say that I already have a working solution, although it isn’t optimal as it must be done manually and offline. I’m hoping that UrBackup will provide a better solution.
My backup requirements are somewhat unusual.
The system in question comprises a 1TB HDD with a primary, hidden and active NTFS formatted 105MB “System Reserved” boot/recovery partition, and the remainder as drive C - a primary, NTFS formatted system partition. The C partition is fully encrypted with Veracrypt, and requires a password at boot to activate.
To be more precise, the disk layout begins with the Veracrypt bootloader, the MBR, the first partition VBR and an NTFS filesystem (System Reserved), followed by a second copy of that partition’s VBR.
What would normally follow would be the second partition VBR, but in this case is just encrypted data. Only the first 512 bytes of that encrypted data are needed for Veracrypt to mount the encrypted partition (but not the filesystem), and map the decrypted version to a virtual device. If a valid filesystem is present on that virtual device then it can be mounted. If not then it can be created.
Under Linux, this Veracrypt virtual device is presented by device mapper as /dev/mapper/veracrypt[n], where “n” is incremented for each Veracrypt device that is mounted (although it’s more technically correct to say “activated”, since no actual filesystem is mounted yet). To Linux, this looks like a regular partition (but, importantly, not like a whole disk).
Similarly under Windows, this Veracrypt virtual device is presented as e.g. \?\Volume{60b38509-f955-11d8-bdab-806d6172696f}\ (using GUIDs) or \device\HarddiskVolume4 (using paths), and then assigned a friendly drive letter, e.g. C. The underlying real partition is seen by the system as a raw, unformatted device, and is not assigned any drive letter.
Every backup solution I’ve tried thus far, both open source and commercial, has failed to correctly restore this system to a pristine working state. The only solution that works is one I devised myself using a combination of other tools. However it isn’t automated, it must be done offline, and I suspect that it results in much larger backups than necessary.
For example, Acronis True Image, for all its sophistication, is quite incapable of restoring this system to a bootable state, even using its “Universal Restore” (to different hardware) feature, and even using Microsoft’s own recovery tools afterwards (bootrec, bcdedit and friends) has no effect.
The problem is that whole disk/partition encryption is necessarily invisible to the OS, and therefore any backup software running within the live Windows environment is incapable of recognising that the partition it’s imaging is actually virtual, and that the characteristics of the real hardware underneath are very different.
Worse still, disk imaging software like True Image gets very confused by such virtual devices, and misinterprets them (and then backs them up) as something quite unusable for restore purposes, so even though the backup process appears to work, the restore will unexpectedly fail, with no possibility of recovery.
To give you an idea of exactly how confused Acronis software gets, I once constructed a rescue meda that combined Acronis Disk Director and True Image with Veracrypt, so that True Image would be able to see the decrypted filesystem. The result was that Disk Director saw the decrypted partition as a “super floppy”, and all of its various partition modification tools were greyed out. True Image refused to display or work with the decrypted filesystem at all.
This is why it’s so important to test your backups … actually test them with a full restore, not merely verify the backup’s checksums.
I find it very disappointing, and somewhat perplexing, that the backup industry has still not developed any strategy for bare metal restore of encrypted storage, despite the exponential growth in public interest in disk encryption in recent years, culminating in Microsoft and hardware vendors building support for it right into the BIOS and OS. This is no longer a corner case, and needs to be more widely supported. Backup should not be viewed as “optional”.
My solution is rather convoluted, but works, and must be run offline using a Linux system (in my case on a USB thumbdrive).
It begins by having the foresight to make a disk image dump (using dd) of the bootsector, MBR and everything up to and including the first 512 (encrypted) bytes of the second partition. This is only 105MB, but is crucial.
(I should also note that the obvious answer of simply using dd to image the whole, raw, encrypted drive doesn’t work, since it end up interpreting some of the random data as bad blocks, and the restored dd is then unbootable. Using ddrescue instead may help, but even then I’d end up with a massive file containing both used and unused sectors, with no easy way to do incrementals, deduplication, or mount individual filesystems within the image.)
I then “mount” (activate) the encrypted partition using Veracrypt, and run ntfsclone on /dev/mapper/veracrypt1, piping the output to a deduplication backing store called “borg”.
Recovery is simply the same process in reverse.
This gives me pristine copies of only the filesystem, which I can then restore manually using ntfsclone, after recreating the partition layout using dd, and activating the encrypted partition with Veracrypt.
However, I dislike having to reboot just to make backups, and I’m not convinced that borg deduplication is as efficient as VHD differentials, either from the perspective of time or data size, especially if the resulting differential VHDs are also combined with a deduplication system.
So, here are my questions, with respect to using UrBackup:
-
If I am always backing up a Windows system partition under Windows, and always restoring it offline using a Linux system, do I actually need to make image backups at all, or can I restore a pristine copy of the NTFS filesystem only, without any partition layout data, using UrBackup under Linux, only from a normal file backup? In other words, can Urbackup make a full and pristine backup of a Windows system partition filesystem using only a file-level backup method in the first place, and if so then does the subsequent restore process under Linux properly handle all properties of this NTFS filesystem, including ADS, junction points, standard file attributes and ACLs, to produce a filesystem that is a pristine copy of the original?
-
Can I have a brief explanation of the deduplication methods employed by UrBackup, and how this compares with bup, fsarchiver, borg, btrfs, ZFS and other deduplication systems? How does the deduplication of image backups compare with deduplicated file backups, in terms of speed and data size?
-
I’ve just completed a UrBackup image backup of my Windows system partition, and one incremental (synthetic) image backup, with the following stats:
Full image backup, Volume: C:, 501 min, 404.38 GB Incremental image backup, Volume: C:, 184 min, 580.58 MB.
How much faster would an incremental backup be using change block tracking? In order to use this feature, do I need to create a new full image backup, after installing the CBT client, or can it start with an existing backup? -
If I stick with image backups, in order to avoid writing invalid partition information (vis-à-vis Acronis True Image and Veracrypt virtual devices), using the UrBackup rescue media and a disk image backup, can Urbackup restore the filesystem only, without making any attempt to recreate partitions, or do I have to mount the VHDz and use something like ntfsclone to make a pristine restore? In that regard, how does one “mount” a VHDz in such a way that it exposes a device node only, rather than a live mounted filesystem, since ntfsclone refuses to operate on live filesystems? Can I do this with qemu-nbd, and how is this complicated by gzip compression and incremental image assembly? Would I need to completely reassemble and decompress the backup to a new file in order to “mount” it, or is the output file merely a handle through which data is piped?
-
Not that I have any articular objection to the UrBackup rescue media, but I prefer using my own, full Linux system to do restores and other maintenance, not least of which because I need an installed copy of Veracrypt. Is there something unique about the UrBackup rescue media that cannot be replicated on my own Linux environment, or can I just use my own system and install UrBackup server and client? Is there any difficultly in then connecting that server to the backup made under Windows, assuming it has access to it?
My instinct tells me that I should just stick to file level backups. I just need to know if a bare metal restore is possible in that case.
Any suggestions or comments appreciated.
) it worked out okay. UrBackup backs up the image unencrypted, it is then restored unencrypted, the TrueCrypt bootloader allows one to skip the encryption during boot, after that I encrypted the drive again.