I am backing up a standalone hyper-v 2016 host with just one internal physical volume and two windows VMs (Windows Server 2016, Hyper-V configuration version 8.0). One VM has one virtual disk, the other has two virtual disks. Whenever I am backing up the VM with two virtual disks, only one disk is backuped up successfully, the other one shows the error “Request of image backup failed. Reason: Opening filesystem on device failed.” Also the hyper-v checkpoint is not deleted after the failed backup.
If I select only one “Volume to backup” via the ‘Image Backup’ settings of the client (f. e. SCSI:0:0 or SCSI:0:1) the backup is sucessful.
Server: Windows 11, UrBackup 2.5.33
Client: Windows Server 2016, UrBackup Client 2.5.23-hyperv
The host has a USB drive connected. It is not used from the operating system / hyper-v.
Just a guess: maybe the client deletes the checkpoint before the second disk is backuped up sucessful.
To further analyse the problem a client debug log would be useful. The client log file is per default located at C:\Program Files\UrBackup\debug.log on Windows. Run C:\Program Files\UrBackup\enable_debug_logging.bat as admin to enable debug logging. Once debug logging is enabled reproduce the problem.
Attach the log files to your new post or send them to bugreports@urbackup.org because they may contain data which should not be public.
I created two virtual volumes (SCSI:0:2 and SCSI:0:3). I setup the client to backup only these two volumes. If I run a backup (full or incremental) the volumes are backuped up twice.
I started one full image backup at 19:40 and one incremental backup at 19:45.
The following message is correct so far: “Image backup is being backed up in a snapshot group together with volumes SCSI:0:X”. It looks like this triggers another backup activity of the other volume, leading to four activities (two initial, and two for the other volumes respectively).
Maybe sometimes this leads to timing problems, and to some of these activities failing.
I also see that while the backup is running one hyper-v checkpoint is created, then deleted, and then a second hyper-v checkpoint is created.
From my point of view this is incorrect behaviour, because on backup should get its data from only one hyper-v checkpoint. Otherwise the backup would be inconsistent.
Thanks for tracking this down to this extent. The cause may be a stray newline in the configuration coming from the client. If you are using a Linux server you can correct it like this for existing clients (requires restarting afterwards):
echo -e "UPDATE settings SET value='ALL' WHERE value='ALL\n' AND key='image_snapshot_groups';" | sqlite3 /var/urbackup/backup_server_settings.db
Will fix it on the server and client side with the next versions.
I updated the backup_server_settings.db file on my Windows system and double-checked it. Then I restarted both the server OS and the UrBackup client service. However, the problems still persist.
I still believe there are multiple backup activities being triggered by different events, and these activities are competing with each other.
Here’s what I think is happening: 1. A backup is initiated.
Two volumes are present: Volume 0 and Volume 1.
Both are added to the job queue — let’s call these Activity A (for Volume 0) and Activity B (for Volume 1).
At this point, the plan is to back up both volumes from the same checkpoint. 2. Activity A (Volume 0) starts first.
A checkpoint is created.
It detects that there’s another volume (Volume 1) in the snapshot group, so it adds another job — let’s call it Activity C — for Volume 1.
However, Activity C is placed before Activity B in the queue, and it doesn’t recognize that a job (B) already exists for Volume 1. 3. Activity A finishes successfully. 4. Activity C starts next.
It doesn’t know about the checkpoint from Activity A, so it deletes the existing checkpoint and creates a new one.
Activity C completes successfully. 5. Now, Activity B starts.
It again detects another volume (Volume 0) in the snapshot group and adds another job — Activity D — for Volume 0.
Sometimes, Activity B succeeds. Other times it fails with the message:
“Request of image backup failed. Reason: Opening filesystem on device failed.”
(Could be a timing issue?) 6. Activity D runs.
Again, it sometimes completes successfully, sometimes not. 7. If Activity B or Activity D fail, the checkpoint is left behind and is neither deleted nor merged.
I’ve tried to trace this behavior in the debug logs, but it’s difficult to map specific log entries to the individual backup activities.
If needed, I can provide debug logs.
Would you recommend installing the backup server on Linux instead?
This is a pretty basic setup — one Hyper-V host with just a few VMs, some of them having multiple virtual disks.
Maybe I’m the only one trying to back up a Hyper-V host using a Windows-based UrBackup server?
These are the logs of one backup - there should be only two activities:
19:40:00 Creating shadowcopy of “hyperv://srv1/SCSI:0:2”…
19:40:27 WARNING: No reference point found. Resetting CBT
19:40:32 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 6
19:40:32 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 50
19:40:39 Deleting shadowcopy for path “C:\VMs\srv1\Virtual Hard Disks\srv1-E.vhdx” -2
19:40:39 Deleting Shadowcopy for dir “hyperv://srv1/SCSI:0:2” 19:40:47 ClientService cmd: #someID#2LOGDATA 1759772427 0-1759772380-Starting unscheduled full image backup of volume “SCSI:0:2”…
3-Image ckup is being backed up in a snapshot group together with volumes SCSI:0:3
19:40:47 Creating shadowcopy of “hyperv://srv1/SCSI:0:3”…
19:40:47 WARNING: Restarting shadow copy of hyperv://srv1/SCSI:0:3 because it was started by this server
19:40:47 WARNING: Restarting shadow copy of hyperv://srv1/SCSI:0:3 because it was started by this server
19:41:16 WARNING: No reference point found. Resetting CBT
19:41:33 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 6
19:41:33 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 50
19:42:14 Deleting shadowcopy for path “C:\VMs\srv1\Virtual Hard Disks\srv1-F.vhdx” -2
19:42:14 Deleting Shadowcopy for dir “hyperv://srv1/SCSI:0:3” 19:42:17 ClientService cmd: #someID#2LOGDATA 1759772516 0-1759772428-Starting unscheduled full image backup of volume “SCSI:0:3”…
3-Image ckup is being backed up in a snapshot group together with volumes SCSI:0:2
19:42:19 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 6
19:42:19 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 50 19:42:31 ClientService cmd: #someID#2LOGDATA 1759772531 0-1759772517-Starting unscheduled full image backup of volume “SCSI:0:3”…
7-Image ckup is being backed up in a snapshot group together with volumes SCSI:0:2
19:42:34 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 6
19:42:34 ERROR: Setting FSCTL_ALLOW_EXTENDED_DASD_IO failed -2. Err: 50
19:42:40 Deleting shadowcopy for path “C:\VMs\srv1\Virtual Hard Disks\srv1-E.vhdx” -2
19:42:40 Deleting snapshot of srv1…
19:42:42 Deleting Shadowcopy for dir “hyperv://srv1/SCSI:0:2” 19:42:47 ClientService cmd: #someID#2LOGDATA 1759772546 0-1759772532-Starting unscheduled full image backup of volume “SCSI:0:2”…
2-Image ckup is being backed up in a snapshot group together with volumes SCSI:0:3
At first, I did it using DB Browser for SQLite v3.13.1 (Windows 64-bit). I double-checked it, but ended up running it again using your code. Let’s see how it goes.
I’m still considering switching the backup server to Linux. I originally chose Windows because I can mount a VHD natively. However, this advantage is lost when using compressed VHDs. That’s another reason why I’m thinking about switching to Linux — under Linux, I can mount compressed VHDs.
