BTRFS performance issues on server

1

One of my servers has about 15 clients, with various file & image backups. The data disk is mounted with

/data btrfs defaults,noatime,nodiratime,compress=zlib 0 2

Over the past few weeks, disk IO has been increasing and performance decreasing. It’s gotten to the point of where even running just 1 client at a time causes severe disk thrashing and painfully slow backup speeds.

I’ve bumped RAM from 8GB to 64GB, but it didn’t help much. Total data size is about 6.5TB. I’ve also noticed that when starting urbackupsrv, the test snapshot process now takes up to a minute, where it would normally be nearly instantaneous:

Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 2.94 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
2198 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % btrfs subvolume snapshot /data/urbackup/testA54hj5luZtlorr494/A /data/urbackup/testA54hj5luZtlorr494/B

What can be done to restore performance?

I’ve combed through a bunch of btrfs performance issue related articles and a few of them mention increasing disk thrashing and significant performance degradation over time, especially with COW. Would a different file system offer better performance/scalability?

2

Here’s an example of the disk thrashing going on, with only 1 client backing up:

image.png1328×516 112 KB

That seems like a lot of writing going on with just 1 active client :confused:

image.png736×231 6.89 KB

The other two backup clients at Indexing… are sitting at “Waiting for filelist” stage…

Total DISK READ : 0.00 B/s | Total DISK WRITE : 1803.76 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 567.12 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
278 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [btrfs-transacti]
936 be/4 urbackup 0.00 B/s 1803.76 K/s 0.00 % 0.00 % urbackupsrv run --config /etc/default/urbackupsrv ~n --pidfile /var/run/urbackupsrv.pid [fbackup load]
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]

3

It needs to go over all files and re-link them during full file backups. Can you disable them?

4

Do you mean disable full file backups and only do incrementals?

5

Yes

6

OK, I’ve disabled full file backups for all clients. Will see how it goes.

Is there any reason/need to do periodic full backups or are incrementals sufficient to ensure all data is backed up?

7

I second this question. While using BTRFS, it seems you keep recommending NOT doing full file or image backups at all? Right now I have fulls for both set to 1 and 1 and your saying if that full gets deleted it will not affect incrementals!? OR we should just disable fulls altogether and begin with incrementals?