I have six machines actively taking backups on my btrfs-backed hard drives. Right after the indexing phase, those hard drives become unresponsive for about a minute: other services (e.g. nginx is logging to the drives, and might fill the logging buffers, blocking until a flush can happen) have to wait. I think this triggers at the point where urbackup is snapshotting the previous backup to create the delta backup.
Are there others that notice this issue? Ideally, this should be fixed at the level of btrfs itself (if that’s indeed the culprit), but I’d be interested in any known workaround.
I have similar issues with btrfs on spinning rust. While to some extent fiddling with sysctls helped it seems that the core issue is that btrfs performs poorly when large numbers of snapshots are present on a volume.
The only way I can keep acceptable performance on my backup host is to turn down the number of active backups kept for each client.
Do you have some references for this? Would be interested in implementing them. Since I’m now running on a 7800X, downtime is only some minutes, if it happens.
I’ll probably tune down the number of kept backups too, then. Would be nice to fix this upstream though!
I don’t have any particular references - just generally reading kernel sysctl documentation and fiddling with things and looking at the durations of the lockups.
# lets try not to be so swap happy
vm.swappiness = 30
# lets not be so quick to throw away dirent/inode cache
vm.vfs_cache_pressure = 50
# more vm tuning to try and wrangle btrfs poor performance
vm.dirty_background_ratio=20
vm.dirty_ratio=40
vm.dirty_expire_centisecs=3000
also using the bfq scheduler instead of the default deadline scheduler:
for i in /sys/block/sd*/queue/scheduler ; do echo "bfq" > $i; done
Keep in mind that your hardware is not the same as my hardware and any tuning you do should be rigorously benchmarked to prove that it is actually helping. I still get lock ups for 20+ minutes at a time during cleanup on an older system so if you are only experiencing a few minutes worth then maybe that is already as good as you’ll be able to get.
