Can't delete incomplete images and can't write to VHD

1

I have 2/10 backups that are failing. I have reviewed some forum posts and reviewed the server log below, and I am not sure if what I am seeing would be resolved by an upgrade from 1.4.14 to 2.0.30, or if that would not help the present issues. This is a server environment where a handful of older ('03), flaky servers have to be backed up; and since the majority of backups are running fine, I kinda wanted to shy away from an upgrade that might (?) break something else. But if I have to upgrade to resolve these issues, so be it. I would appreciate your thoughts! Thank you again for the awesome product!

blob.jpg985×362 180 KB

blob.jpg886×144 39.5 KB

blob.jpg882×187 57.4 KB

2016-12-13 03:35:39: WARNING: Deleting incomplete image "\\wen-syn-001\Backups\urbackup\WENSVR_02\Image_SYSVOL_161212-0426.vhd" failed. 2016-12-13 03:35:39: WARNING: Deleting incomplete image "\\wen-syn-001\Backups\urbackup\ALHS-RDS2\Image_SYSVOL_161212-1902.vhd" failed. 2016-12-13 03:37:52: ERROR: Error opening file '\\wen-syn-001\Backups\urbackup\WensvrEHR\Image_D_161017-0211.vhd' 2016-12-13 03:37:52: ERROR: Error opening file '\\wen-syn-001\Backups\urbackup\WENSVR_04\Image_D_161024-0440.vhd' 2016-12-13 03:37:53: ERROR: Error opening file '\\wen-syn-001\Backups\urbackup\WensvrLab\Image_D_161117-1903.vhd' 2016-12-13 03:37:53: ERROR: Error opening file '\\wen-syn-001\Backups\urbackup\ALHS-RDS2\Image_C_161123-1903.vhd' 2016-12-13 10:00:39: ERROR: Pipe to client unexpectedly closed has_error=true 2016-12-13 10:00:45: ERROR: Backup failed 2016-12-13 10:00:45: WARNING: Exponential backoff: Waiting at least 21h 20m before next image backup 2016-12-14 02:32:41: WARNING: During getting MBR: Dynamic volumes are not supported. It may work with mirrored whole disk volumes though. Volume: 'C:' 2016-12-14 02:39:44: WARNING: During getting MBR: Dynamic volumes are not supported. It may work with mirrored whole disk volumes though. Volume: 'D:' 2016-12-14 03:01:04: WARNING: During getting MBR: Dynamic volumes are not supported. It may work with mirrored whole disk volumes though. Volume: 'G:' 2016-12-14 03:46:07: WARNING: Deleting incomplete image "\\wen-syn-001\Backups\urbackup\ALHS-RDS1\Image_SYSVOL_161213-0431.vhd" failed. 2016-12-14 03:46:08: WARNING: Deleting incomplete image "\\wen-syn-001\Backups\urbackup\WENTS01\Image_SYSVOL_161214-0232.vhd" failed. 2016-12-14 03:46:09: WARNING: Deleting incomplete image "\\wen-syn-001\Backups\urbackup\WENSVR_02\Image_SYSVOL_161214-0232.vhd" failed. 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: WARNING: Reading from parent hash file failed (May be caused by a volume with increased size) 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:15: ERROR: Writing to file failed 2016-12-14 03:48:15: ERROR: Last error: 64 2016-12-14 03:48:16: ERROR: Writing to file failed 2016-12-14 03:48:16: ERROR: Last error: 64 2016-12-14 03:48:16: ERROR: Writing to file failed 2016-12-14 03:48:16: ERROR: Writing to file failed 2016-12-14 03:48:16: ERROR: Last error: 64 2016-12-14 03:48:16: ERROR: Last error: 64 2016-12-14 03:48:16: ERROR: Writing to file failed 2016-12-14 03:48:16: ERROR: Last error: 64 2016-12-14 03:49:18: ERROR: FATAL: Error writing to VHD-File. 2016-12-14 03:49:18: ERROR: FATAL ERROR: Could not write to VHD-File 2016-12-14 03:49:18: ERROR: FATAL: Error writing to VHD-File. 2016-12-14 03:49:18: ERROR: FATAL: Error writing to VHD-File. 2016-12-14 03:49:18: ERROR: FATAL ERROR: Could not write to VHD-File 2016-12-14 03:49:18: ERROR: FATAL: Error writing to VHD-File. 2016-12-14 03:49:18: ERROR: FATAL ERROR: Could not write to VHD-File 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: FATAL: Writing failed after cleanup 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: FATAL: Writing failed after cleanup 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: FATAL: Writing failed after cleanup 2016-12-14 03:49:18: ERROR: Error writing footer 2016-12-14 03:49:18: ERROR: FATAL: Writing failed after cleanup 2016-12-14 03:49:19: ERROR: Backup failed 2016-12-14 03:49:19: WARNING: Exponential backoff: Waiting at least 40m before next image backup 2016-12-14 03:49:21: ERROR: Backup failed 2016-12-14 03:49:21: WARNING: Exponential backoff: Waiting at least 40m before next image backup 2016-12-14 03:49:22: ERROR: Backup failed 2016-12-14 03:49:22: WARNING: Exponential backoff: Waiting at least 10h 40m before next image backup 2016-12-14 03:51:44: ERROR: Backup failed 2016-12-14 03:51:44: WARNING: Exponential backoff: Waiting at least 40m before next image backup 2016-12-14 03:56:57: ERROR: Error opening file '\\wen-syn-001\Backups\urbackup\WensvrEHR\Image_D_161024-0353.vhd' 2016-12-14 03:56:57: ERROR: Error opening file '\\wen-syn-001\Backups\urbackup\WensvrLab\Image_D_161117-1903.vhd' 2016-12-14 03:56:57: ERROR: Error opening file '\\wen-syn-001\Backups\urbackup\ALHS-RDS2\Image_C_161123-1903.vhd'

2

I would say yes to the upgrade, but at the same time you have '03 Servers, and from what I remember you want at least '08 when you upgrade.

It seems like they were working before, judging by how one of them was only 9-ish days ago; did something change for WENSVR_04 9 days ago, and did something happen to WensvrEHR about a month ago?

Two things I would suggest, is:

  1. Manually run the cleanup script on your server.
  2. Manually run the remove_unknown script on your server.

Also, is your drive space possibly full?

3 
It seems like they were working before, judging by how one of them was only 9-ish days ago; did something change for WENSVR_04 9 days ago, and did something happen to WensvrEHR about a month ago?

Not that I’m aware of. I’ll try your other suggestions before I investigate this avenue further.

Two things I would suggest, is:

Manually run the cleanup script on your server.
Manually run the remove_unknown script on your server.
Also, is your drive space possibly full?

The repository (Synology NAS) did indeed fill up a while back, within the past couple weeks. I just started the “cleanup 0%” command and I noticed this:

Pasted image996×336 48.8 KB

Is it OK to manually remove that file? Or should I do something else instead?

After this I will also run the remove_unknown.bat script.

4

Looks like I have some other warnings and errors too:

Pasted image958×658 64 KB

I got the same bottom set of four errors when I ran the remove_unknown.bat script.

When I ran a search on the repository in those agent folders, the only files I found ending in the last four digits were vhd.hash files. No actual VHD files existed.

5

It looks like those were just image attempts that failed and were never cleared up.

Cleanup and/or remove_unknown should remove the file automatically for you.

6

Thanks, I think the scripts may have fixed the problem. Full backups at about 50% now. Will update later.

7

Well, between finding and removing really old backups that had not been removed with retention and using the scripts, the issue has been resolved and all the agents are backing up fine! Thank you for the help!

8

hey - even though you solved your issue i had a similar problem a while back.

Synology NAS as Storage aswell.
Soloution for us, was to check and repair the Filesystem as it was damaged somehow.
(Few days later first one and then a two days later another disk crashed - maybe that was connected. but hey, The Raid 6 survived).

Maybe you should check the FS and disks SMARTs aswell - just to make sure everything’s alright.