Slow backup of a lot of files (200K-300K files)

Server
1

Hi,

i have “slow, never ending” issue with UrBackup Server 2.2.7.

One of clients starts to backup with acceptable performance (for example 20-50Mbps on 100Mbps - its OK for me). But after transfering for example 200GB slows down and hashes and linking small file fof few minutes.

Please look to part of client log:
"

28.02.18 10:22 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-06 mocowanie silownika.psm”
28.02.18 10:22 DEBUG PT: Hashing file “profil 80x80 poz prz dach.ipt”
28.02.18 10:22 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-06 mocowanie silownika_Blacha gieta.dxf”
28.02.18 10:22 DEBUG Saved metadata of 213083 files and directories. 49% done…
28.02.18 10:22 DEBUG PT: Hashing file “profil 80x80 poz prz dol.ipt”
28.02.18 10:22 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-06 mocowanie silownika_Blacha rozwinieta.dxf”
28.02.18 10:23 DEBUG PT: Hashing file “profil 80x80 poz prz gora.ipt”
28.02.18 10:23 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-07 mocowanie silownika_odb.dft”
28.02.18 10:23 DEBUG PT: Hashing file “profil 80x80 poz prz sr.ipt”
28.02.18 10:23 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-07 mocowanie silownika_odb.pdf”
28.02.18 10:23 DEBUG Saved metadata of 213085 files and directories. 49% done…
28.02.18 10:24 DEBUG PT: Hashing file “profil 80x80 poz sr sr.ipt”
28.02.18 10:24 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-07 mocowanie silownika_odb.psm”
28.02.18 10:24 DEBUG PT: Hashing file “profil 80x80 poz tyl dol.ipt”
28.02.18 10:24 DEBUG PT: Hashing file “profil 80x80 poz tyl gora.ipt”
28.02.18 10:24 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-07 mocowanie silownika_odb_Blacha gieta.dxf”
28.02.18 10:24 DEBUG PT: Hashing file “profil 80x80 poz tyl sr.ipt”
28.02.18 10:25 DEBUG PT: Hashing file “profil 80x80 skos prz lew.ipt”
28.02.18 10:25 DEBUG PT: Hashing file “profil 80x80 skos prz pr.ipt”
28.02.18 10:25 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-07 mocowanie silownika_odb_Blacha rozwinieta.dxf”
28.02.18 10:25 DEBUG PT: Hashing file “profil 80x80.ipt”
28.02.18 10:25 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-08 mocowanie zaworow.dft”
28.02.18 10:25 DEBUG PT: Hashing file “stopa.ipt”
28.02.18 10:25 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-08 mocowanie zaworow.pdf”
28.02.18 10:25 DEBUG Saved metadata of 213089 files and directories. 49% done…
28.02.18 10:25 DEBUG PT: Hashing file “104040.stp”
28.02.18 10:25 DEBUG PT: Hashing file “104080.stp”
28.02.18 10:25 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-08 mocowanie zaworow.psm”
28.02.18 10:26 DEBUG PT: Hashing file “108080.stp”
28.02.18 10:26 DEBUG PT: Hashing file “303876.stp”
28.02.18 10:26 DEBUG HT: Linked file: “/media/ur/192.168.5.105/180224-0109/data/NAS/TMA/TMA_KONST_TEMP/KONCEPCJE/koncepcja Jokey/02 - czesci uzyte/01 mechanizm poziomy odbioru/01-02 - Blachy stalowe/01-02-08 mocowanie zaworow_Blacha gieta.dxf”
28.02.18 10:26 DEBUG Saved metadata of 213093 files and directories. 49% done…

“Saved metadata of 213093 files and directories. 49% done”

Is this so slow because of number of files?

Below my settings of Server/Client:
urb log.txt (4,3 KB)

1
11175×217 22.2 KB
3
31145×672 54 KB
2
21176×691 58.9 KB
4
4648×828 60.6 KB

Client and server are in the same network.
I tried with datto snapshot and without.
I use client downloaded and preconfigured from server.

Can any body please help me to resolve that isue?

2

I have forgot to tell that server is running on Centos 7, client on Ubuntu 14.04 LTS

3

Hi

You can look at virtual clients in the docs/forums, to understand how to set it.
Possibly, backup the 250gb using a few different virtual clients, so each part of the backup is smaller.

You can also for that client only ‘enable specific settings for this client"’ , then in advanced tab , switch all backup mode to raw.

4

Hi, Virtual Client do the work (great feauture - I didnt know about it) but still it seems that urbackup isn’t using full resources. HDD/Network and CPU are bored on server and backup client.