Suddenly this task has become much slower than it used to be. What would cause this? I’m trying to see where to troubleshoot. Could it be the BTRFS file system being backed up to? (70% full, raid10, seperate drives from server) Is it likely the server where UrBackup server and the file list database is located? (slow hard drive, not enough memory or cpu?) Could the file database just have gotten way too big? What exactly is occurring during this step? Thanks.
Server: 2.2.11 (ubuntu server 16.04 and latest server 18.04 with latest kernel)
clients: all 2.2.6
replaced server (2.3.7 with 2.3.4 mixed clients of mac,windows,linux) and this has creeped up again… backups start our super fast when server is first setup (1-5 minute file backups), then it seems once I reach maybe too many clients or maybe backups… “syncing file system…” step takes forever and we’re up to like 30 minute incremental backups. There is currently 352 clients connected and only keeping 20 incremental backups and 2 fulls. I’m just looking for what the bottleneck would be here. Is it the BTRFS backup volume? (55% full) or the drives where the urbackup database is located? Thanks.
I haven’t looked at the btrfs code, but I think the problem is the btrfs syncing isn’t “fair”, so new writes on the fs can delay syncs indefinitely. I have recently even seen the same issue on the linux-block mailing list with ext4 ( https://www.spinics.net/lists/linux-block/msg38204.html ). You could try raising this issue on the linux-btrfs mailing list.
As with every congestion problem the two main solutions are either increasing capacity (probably IOPS of your storage) or decreasing usage (probably number of simultaneous backups).
Is it maybe an idea to pause other running backup threads until the filesystem sync is done? I think that slows backups less down than the current situation.