Incremental backup traffic and io

Server
1

Hello @ all,

I am new to urbackup so please excuse my “dummy” questions.

I have istalled the urbackup 2.0.38 server on ubuntu 16.04 and have a urbackup client on centos.
Then I configured incremental backups with the flags:

encrypted transfer
compressed transfer
Calculate file-hashes on the client

As I have many files on the client (approximately 5 million) I though with incremental backup only the changed files will be transfered to the server and cause IO to it. Unfortunately it seems to transfer all files to the server to check if they where changed. This is causing much traffic and disk IO and takes also more then 10h.

How to be able to setup the client to check the filechanges localy, and then just transfer the changed files to the server?

Thank you very much

Regards
Andreas

2

Try without compression, specialy if you re on the LAN.
A full takes 10h and and in incremental takes 10h?
If you run windows and have a lof of small files; try a volume/block based backup instead of files.

3

Hello orogor, first of all thx for your answer!

I have now disabled compression. But I though that the compression will be made on the client and then transfered to the server. This is why it should cause less traffic.

In the activity log i can see that one File takes arround a second. This is quite to long for a small jpg file.

03.01.17 08:58 DEBUG PT: Hashing file "hauptschalldaempfer-auf-serienanlage.jpg" 03.01.17 08:59 DEBUG Loading file "bastuck-adapter-hauptschalldaempfer-auf-serienanlage-auf-r-455.jpg" 03.01.17 08:59 DEBUG Loading file "md" (metadata only) 03.01.17 08:59 DEBUG PT: Hashing file "bastuck-adapter-hauptschalldaempfer-auf-serienanlage-auf-r-455-mm.jpg" 03.01.17 08:59 DEBUG Loading file "73876" (metadata only)

For me it seems that the metadata is loaded form the server? Is this correc?

Regards
Andreas

4

try to unmark
Run backups with background priority on the clients:

5

Hi reine, this setting is not active.

6

Yes, that’s how it work, but:
if you are on LAN; compression is cpu intensive and slow down backup,
if you are on wan/slow link,it will help because you ll transfer less data
(at home i upload at 50Ksec= compression, office i upload at 1M/sec = compression disabled)

We d need to see more lines or see ms to make sure.
Maybe it s 08:58 and 999ms , then 08:59 and 001ms

I think too. Only transfering metadata takes 10h?
Maybe try to backup less, or split the backup and use virtual client, with different backup frequencies.

7

Ok, now I have done a few tests on the backups. I tried now to do a backup on another centos server. The client calculated that there are just 160MB of data (incremental data) and started to copy. After a couple of seconds (at arround 60MB) it stopped copying (no progress in the activity tab). I checked then the urbackup server VM and saw that it causes for some reason very very much IO and arroung 50% CPU usage to the VM node.

I realy do not understand why. I have disabled compression. But the server seems to try to do something on the filesystem. Something very large. And I don’t know what or why?

I have write speeds at arround 100-200 MB/s on my HDDs so this cannot be the problem.

Does anyone have an advice for me. Would really appreciate it.

Thank you very much.