Backup speeds is very slow

Hi,

I am running UrBackup Server v2.3.8.0 on Debian Buster. All the clients are running UrBackup Client Controller v2.3.4.0 on Debian Buster as well except one windows client.
The backup speed is extremely slow on the Debian clients. currently a full file backup job of 133GB is running for an hour and has achieved 400MB and it needs more than a month to complete. The ETA on the Web interface:

I used to use rsnapshot which uses rsync and a full backup took for this client around 2-3 hours.
The backup server is a Ganeti LVM instance and the backup directory is a nfs mount directly from the Ganeti master (host). It is one Western Digital Red 2TB Sata 6GB/s 128MB cache and 5400Rpm . This is the same NFS mount rsnapshot wrote to.
I tested the network speed:

I have unchecked the run with background priority box, but it is not getting better.

Thank you for the help.

Something new about this. Can I do anything to improve the speed?

If I understood this correctly the speed refers to the write operation on the disk. If so then this is really slow.
dd can write a 1GB file to the same directory in in 10 seconds.

If the speed refers to the network speed it was constant at 17 Mbits/sec for the whole time without any drops.
The speed on Urbackup server under Activities changes continuously between 10 Kbits/sec and 14 Mbits/sec.

I hope you can give me any hints to get this work.

Thanks

After further investigation I have the following:
network speed between urbackup LVM Ganeti VM and Ganeti node (host):

  • iperf urbackup as client:
iperf -c 10.0.10.10
------------------------------------------------------------
Client connecting to 10.0.10.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.10.11 port 42674 connected with 10.0.10.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  22.6 GBytes  19.4 Gbits/sec
  • iperf on Ganeti server:
iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.0.10.10 port 5001 connected with 10.0.10.11 port 42674
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  22.6 GBytes  19.4 Gbits/sec

network speed between urbackup and a backup client:

  • iperf on urbackup as server:
iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.0.10.11 port 5001 connected with 10.0.10.150 port 44400
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.8 sec  13.1 MBytes  10.2 Mbits/sec
  • iperf on backup client:
iperf -c 10.0.10.11
------------------------------------------------------------
Client connecting to 10.0.10.11, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.10.150 port 44400 connected with 10.0.10.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec  13.1 MBytes  10.8 Mbits/sec

The speed varies between 10.8 Mbits/sec and 14.9 Mbits/sec on different tries.

dd from urbackup server to the NFS share which is the backup location:

dd if=/dev/zero of=/backup/test bs=1GB count=10 oflag=dsync
10+0 records in
10+0 records out
10000000000 bytes (10 GB, 9.3 GiB) copied, 250.428 s, 39.9 MB/s

So it is definitely not an I/O issue of the NFS share. It is obvious that the backup client 10.0.10.150 has a low bandwidth of max 14.9 Mbits/sec, but this doesn’t explain the Kbit/s speed on the Activities view on urbackup server when backing up this client.

I hope this information will help solving this issue.

Thanks

Unfortunately I don’t have the benchmark of iftop any more, but it sowed exactly the same speed as on Urbackup while backing up to the NFS share.
I made a test of backing up the the same client to the local disk of Urbackup and iftop showed the ~ 14 Mbits/sec as well as Urbackup.
So this seems to be related to the combination of backing up a client to the NFS share through Urbackup. Thus the IO of the NFS share without Urbackup and the network speed between client and Urbackup are at the max speed. only the combination between them has this drawback of the backup speed.

I know this is 19 day olds but I just have to comment on this. And 3x2b2ff5, I am not trying to single you out here, it’s just, why do you guys think this will work at all well? Think about it. Your urbackup server is reaching out over the NIC connection to a server to back it up, transferring data to the urbackup server, then you ask it to write that data over the same NIC to another server. There is obviously going to be contention on that link, the data in and the data out cannot travel simultainiously.

I use iSCSI storage for all my urbackup servers but I see no reason why NFS would not work but I don’t know how to force an NFS mount to use a different interface but I’ m guessing it is possible. Point is I always use a seperate interface to connect my storage, seperate from the interface urbackup uses to access the LAN to pull backups. On physical servers some will say “I don’t have the option of adding another NIC”, well you have to ask yourself, if that is not physically possible (not just financially) then adding backup storage directly to your server is the other option.You have to make choices. If this is for your business you have to go to management and say “Backups are taking too long, we are at risk, here is how we mitigate that risk”

Take this as you will but I’ve been doing server backups for close to 30 years and this had been the best solution to improving backup performace to remote storage.

BTW, urbackup is the most stable, reliable backup system I have ever used. Even better than those multi $1K systems I’ve been forced to use in the past.

Usually UrBackup is slowed down by random IO. NFS is worse than iSCSI (I guess) because iSCSI at least enables use of the local in-memory (writeback) Linux block cache. You can also use e.g. btrfs with iSCSI. The hard-linking operations are sequential (per backup) in UrBackup and thus the network latency matters there. Of course having other stuff using the same network interface increases the NFS latency.