Slow FILE_METADATA loading. Backups Never ends approaching 100%

Ciao,
I inform that I also I use Gravity Zone on all clients …

I have also BitDefender Gravity Zone in my network, with the same problems.
It is strange, because from the begining I used BitDefender without problems, aprox from the last 3 months is the problem present.
Windows update??? :smiley:

Unfortunately I can confirm that I’m using Bitdefender Gravity zone too and I cannot do anything about it.By the way it is up and running since at least a couple of years and it was living happily with urbackup…so I don’t know what is the problem and why now…

Could (s)one of you please do the client debug logs/memory dump/procmon trace thing I requested? Maybe this can be fixed on the UrBackup side (If Bitdefender just hangs, it is probably not going to be fixed – a memory dump would tell if this is the case).

@uroni if you could give me some instructions on how to do what you requested, i would be happy to.

For debug log (least likely to be useful) see Having problems with UrBackup? Please read before posting for instructions.

For procmon capture see e.g. Basic Steps for Making a Process Monitor (ProcMon) Capture - Microsoft Community Hub .

For memory dump see e.g. %kb_name - %short_descr - Knowledge Portal .

Hi guys
@antonis did you provided any file? Any news? I’ m trying to figure out if I can set some exclusion rules on gravity zone to make the urbackup process to finish safely…but without any luck until now.
Someone already tried this way?@uroni do you think it could work?
Thanks
Andrea

We are having the same kind of issue on a couple of clients.

We did a test yesterday, as we are also using Bitdefender, we uninstall bitdefender on one client to test and get the same issue.

I also test to completely uninstall the client and reinstall, same issue again.

there’s no link between the client version as we are having both, working and non-working for the same version.

It’s not related to Windows version, as we are having again, working and non-working for the same Windows version.

Another interesting thing is, windows clients that are still “pending”, I got a lot, a lot of CLOSE_WAIT using netstat -an that are going to the IP of the server using port 55415

Windows’s client firewall is also disable

Has this issue been reported to Bitdefender? It needs to be fixed on their side.

When I looked at the memory dump sent to be it was hanging while opening a file ( CreateFileW ).

@uroni I did some test yesterday using one Windows server that we uninstall Bitdefender and got the same issue!

For our problems, we are having 3 scenarios

1- If it can start the backup, it will hang at 100% and I got a lot of “Loading FILE_METADATA” in the logs, if I restart urbackupclient, it will “end” the backup but with a couple of error about some missing METADATA

2- OR worst, it never start the backup and in the log we have “Connecting for filelist (async)…”

3- everything works as normal :wink:

But I can confirm that it’s not related to Gravity zone at all and not even related to Windows Firewall.

As our testing, I also create a complete NEW urbackup server just to make sure it’s not related to our actual config and/or our actual urbackup database and I got the same issue as #1 on this machine too.

Another thing related to our scenario #1

In the server log I got some

Established internet connection. Service=1
and
Established internet connection. Service=0

sounds like the client is trying to reconnect and it can’t. and at this stage, the logs on the backup job will loop about loading METADATA. If I restart the client it will end and the backup will be good except some error about metadata missing

as you can see in the log:

|01/24/24 14:47 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6.20312 MB at 67.528 KBit/s|
|—|—|—|
|01/24/24 14:49 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 48 Bit/s|
|01/24/24 14:50 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6.03125 MB at 3.8 KBit/s|
|01/24/24 14:52 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 48 Bit/s|
|01/24/24 14:53 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6.03125 MB at 3.8 KBit/s|
|01/24/24 14:55 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 48 Bit/s|
|01/24/24 14:56 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6.03125 MB at 3.8 KBit/s|
|01/24/24 14:58 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 48 Bit/s|
|01/24/24 14:59 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6.03125 MB at 3.8 KBit/s|
|01/24/24 15:01 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 48 Bit/s|
|01/24/24 15:02 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6.03125 MB at 3.8 KBit/s|
|01/24/24 15:04 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 48 Bit/s|
|01/24/24 15:07 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 1.296 KBit/s|
|01/24/24 15:10 |DEBUG |Loading urbackup/FILE_METADATA|HpBjz1Icoeir6xVuXGTC|4. Loaded 6 MB at 1.296 KBit/s|
|01/24/24 15:10 |ERROR |Error getting file metadata. Errorcode: CANNOT_OPEN_FILE (3)|
|01/24/24 15:11 |ERROR |Error informing client about metadata stream end. Errorcode: TIMEOUT (2)|
|01/24/24 15:11 |INFO |Waiting for file hashing and copying threads…|
|01/24/24 15:11 |INFO |Writing new file list…|
|01/24/24 15:11 |DEBUG |Symbolic links created.|
|01/24/24 15:11 |INFO |Transferred 7.20487 MB - Average speed: 40.312 KBit/s|
|01/24/24 15:11 |INFO |(Before compression: 22.2105 MB ratio: 3.08271)|
|01/24/24 15:11 |INFO |58.0523 GB of files were already present on the server and did not need to be transferred|
|01/24/24 15:11 |DEBUG |Script does not exist urbackup/post_incr_filebackup|
|01/24/24 15:11 |INFO |Time taken for backing up client vDCVAULTFS.10ruptiv.local: 25m 20s|
|01/24/24 15:11 |INFO |Backup completed with issues|

At 15:11 I restart the client and the job continue and end correctly.

Have you followed John McAffee’s instructions on how to uninstall your virus scanner?

1 Like

@uroni and it’s not windows related on the client side, 'cause I have also some Linux (debian) that can’t backup too with the same behaviour #2

Hi,
Server: 2.5.32 on Ubuntu 20.04, loglevel=info.
Client: 2.5.25 on Windows 10Pro with GravityZone.


@uroni we did a bunch of test and now having a 100% working environment. But I think that, we would need a little modification in urbackup server :wink:

First, I confirm that, the never ending backup was 'cause of something in Bitdefender Gravity related to the “process” or urbackup. We needed to whitelist the urbackupclient process in Gravity and everything works after that!

We also found out that sometime if the FULL backup is very huge at some point it will timeout but it will recover properly on the next incremental.

Another thing, we did a bunch of test about the other bug a lot of people are having about the data corruption in the database that the repair can’t fix. After doing a couple of test I can confirm that right away if the disk/partition of the database got full, it can’t be repair and you are stuck with a data corruption and all the “Backup” after that point will no be 100% working properly and even the night job will have a hard time.

That being said, is it possible to have something in the server process that will stop ANY job of backup and nightly job IF the free space is not enough with maybe a custom option in the settings with a % of free space ?

'cause of that issue with free space that we got a long long time ago we had to start a completely fresh new backup server last week after doing a lot of test to isolate everything and have a nice working setup.

Hope this post will help others users too!

Risolto temporaneamente creando un task giornaliero che riavvia il servizio UrbBackup sui Client

Could you explain to me how to put urbackup on the gravity whitelist? thank you so much

I did it from Policies->Select the policy-> Antimalware->Setting and there in-policy exclusions but it didnt seem to work for me.

@PointPubMedia Could you please confirm you did it this way or is there a better one?

Hy,
My update:
Scenario1: Urbackup server (2.5.32) on Ubuntu in LOCAL NETWORK only with Active Directory + Gravity Zone + limited users = working perfect !!!
Scenario2: Urbackup server (2.5.32) on Ubuntu in the cloud, internet mode only: clients with Gravity + clients with BitDefender Free = errors randomly.

You need to exclude the process C:\Program Files\UrBackup\UrBackupClientBackend.exe

2 Likes