First incremental file backup after full backup taking a long time

Server
1

I’ve got an Ubuntu 22.04 server with around 14TB of data in 7 million-ish files being backed up by UrBackup 2.5.28 running on a TrueNAS system. The first full file backup a few months ago took several days, but after that the incrementals have taken around 45 minutes to run.

It did another full file backup starting on 6/18 and completing about two and a half days later. A few hours later it started an incremental, and it looks like that’s been running for the last 15+ hours.

The client log hasn’t done anything but “Looking for old Sessions… 0 sessions” for most of that time, and the log of the running backup is spitting out a few lines like this every few seconds:

06/21/23 13:40 DEBUG Skip phash for id 1547345 PFOcf2BbH1HVwEp9rveLV878D1xwl0BJZURFHnFM5knDKLIe3oS6K1hYHSCooxwuviV1DKLkyLhq0PMt0D3Yxw

In /usr/local/var/backup on the client these are the most recently changed files:

-rwx------ 1 root root 617508864 Jun 21 06:49 backup_client.db
-rwx------ 1 root root 135033032 Jun 21 13:40 backup_client.db-wal
-rwx------ 1 root root 425984 Jun 21 13:40 backup_client.db-shm

And in data there’s this:

-rw-r–r-- 1 root root 352915957 Jun 20 22:10 filelist.ub

“urbackupclientctl status” on the client shows me this:

“internet_connected”: true,
“internet_status”: “connected”,
“last_backup_time”: 1687308530,
“running_processes”: [{
“action”: “INCR”,
“done_bytes”: 6260523736,
“eta_ms”: 0,
“percent_done”: 67,
“process_id”: 97,
“server_status_id”: 43,
“speed_bpms”: 0,
“total_bytes”: 9386488398

Where should I look for further troubleshooting info? Should I just never do full file backups of this machine? Thanks!

2

Excluded files: /dev/;/sys/;/proc/;/run/;/media/*;/swap.img;/mnt;/dev;/sys;/proc;/run;/media;/var/log/journal

Default directories to backup: /|rootfs/follow_symlinks,symlinks_optional,share_hashes,one_filesystem;/mnt/storage;/boot

/mnt/storage is where 99% of the files are.

3

I’m using btrfs myself where this isn’t necessary, but my guess would be that it is the moving of large unchanged sub-trees to .directory_pool (assuming you are not using zfs snapshots).

You could confirm this by e.g. strace -p (if there is an equivalent on FreeBSD).

4

Hah, well, I’m not sure, because it looks as though the act of using ‘truss’ (similar to strace) to inspect the running urbackupsrv process killed it, which isn’t something I’ve seen happen before.

However, your “assuming you are not using zfs snapshots” has pointed me to the part of the documentation about using zfs snapshots on the server. So I think my next step may be to rebuild this with the zfs dataset delegated into the urbackup jail so I’m using zfs snapshots; I hadn’t realized until now that that was an option.