Backup freeze at 100%

Server
1

Hello,
We have a urbackup who is hosted on standalone ubuntu server 20.04 LTS
All is working fine until the hypervisor that hosts the production VM crash.
I think that the impacted VMs are the ones that should certainly have been backing up at the time of the crash.
Backups on some servers remain stuck at 100%
I restarted the service urbackupsrv and desinstall/reinstall client software on VM (Windows2019) but that did not resolve the problem.
I checked the disk space on server and on Client and there is enough space left.

Does anyone have a solution please?

Below are the last urbackup logs :
2023-12-27 15:08:59: ERROR: driver returned an error performing specified action. check dmesg for more info: No space left on device
2023-12-27 15:28:56: ERROR: Server admin stopped backup.
2023-12-27 15:43:04: ERROR: Connecting to ClientService of “SRVFIC01_grad” failed: Removing snapshot on “SRVFIC01_grad” for path “D” failed
2023-12-27 15:43:04: ERROR: Timeout during sending metadata stream end (1)
2023-12-27 15:43:04: ERROR: Error informing client about metadata stream end. Errorcode: TIMEOUT (2)

image
image901×104 29 KB

3

Check the “show log” next to the backup at 100% and post back when the last 5 lines say here.

We run this on QNAP NAS directly and the slower NAS (no SSD and a close CPU) can take 2-3 hours to create all the links on the server, however the data has been collected from the client, server is building the restore views, it takes that long (on a slower machine)

4

Hello,
Below the last 5 logs from the backup of a VM which started on January 16 and which is still blocked at 100%.
Urbackup is installed on a high-performance server with a Xeon processor.
The server was installed in August and everything worked well at first. Backups were very fast. But today it is blocking at 100%.

Please help !

|16/01/24 12:17 |DEBUG |Loading urbackup/FILE_METADATA|rnj15ivL2CHmvlsSsAHd|2501. Loaded 15 MB at 8 Bit/s|
|—|—|—|
|16/01/24 12:18 |DEBUG |Loading urbackup/FILE_METADATA|rnj15ivL2CHmvlsSsAHd|2501. Loaded 15.3917 MB at 54.68 KBit/s|
|16/01/24 12:20 |DEBUG |Loading urbackup/FILE_METADATA|rnj15ivL2CHmvlsSsAHd|2501. Loaded 15 MB at 8 Bit/s|
|16/01/24 12:21 |DEBUG |Loading urbackup/FILE_METADATA|rnj15ivL2CHmvlsSsAHd|2501. Loaded 15.3917 MB at 54.688 KBit/s|
|16/01/24 12:23 |ERROR |Error getting file metadata. Errorcode: TIMEOUT (2)|

5

Where is this log from?

6

From here :

7

It says timeout, so this looks like the client system gave up, normally if it goes offline (cant ping it) it says it went offline, however others here might be able to verify that

8

Effectively the client system gave up and this happens randomly when performing full backups, both on newly installed servers and on old installations that have UrBackup version 2.5.32

9

All clients that are affected by this have the same behavior that they report in this thread: