It seems like my server never automatically deletes any backups during the cleanup window. I have global quota set to 90% and e.g. a maximum of 14 daily incremental backups. Yet, I have hundreds of incremental backups for each client.
It took almost a year for my 8x 10TB raidz2 to fill up; I was waiting to see if the server would start deleting backups when the quota is hit. Now I’m at 1% free (reported by zfs list) and the server does emergency cleanups during normal backup operation. That does not work reliably as zfs takes some time to reclaim space after deletions. Manually stopping the server and running ‘urbackupsrv cleanup’ works as expected.
The backup window is at the default 1-7/3-4. Is there anything I could have misconfigured? Is there a way to enforce the maximum number of backups to keep? It’s quite possible that the server has problems checking the available space on the backup volume, if it is using the posix interface (see output of df below):
root@backup:~# zfs list tank/urbackup/images
NAME USED AVAIL REFER MOUNTPOINT
tank/urbackup/images 1.61T 82.2G 205K /tank/urbackup/images
root@backup:~# zfs list tank/urbackup/files
NAME USED AVAIL REFER MOUNTPOINT
tank/urbackup/files 48.1T 82.2G 205K /tank/urbackup/files
root@backup:~# zfs list tank
NAME USED AVAIL REFER MOUNTPOINT
tank 49.9T 82.2G 56.0M /tank
root@backup:~# df -h /tank/urbackup/images
Filesystem Size Used Avail Use% Mounted on
tank 83G 56M 83G 1% /tank
root@backup:~# cat /etc/urbackup/dataset
tank/urbackup/images
root@backup:~# cat /etc/urbackup/dataset_file
tank/urbackup/files
root@backup:~# cat /etc/urbackup/backupfolder
/tank/urbackup/backupfolder
Edit:
Manually stopping the server and running ‘urbackupsrv cleanup’ works as expected
Well, i ran urbackupsrv cleanup -a 20% and it removed a few TB. But I still have more than the configured maximum backups for some clients.
You are right, the problem is settings made via group are not applied to the individual clients. When I change a settting in the group, it does not update the client settings.
The output of df is not useful to get the free space of btrfs and zfs systems. Can you consider intergrating the appropriate btrfs and zfs tools? Or alternatively, make it possible to call an external script.
I stumbled across another problem; When the server deletes a backup the corresponding zfs dataset is not deleted. I didn’t notice before because I thought full backups were being kept as the base for incremental backups.
My server is already logging with debug level, yet I can’t find any log output relating to the cleanup process. I am aware the zfs support was contributed by a third party; If you don’t have the time to look into this please refer me to them. With a little bit of help I might be able to start working on fixes myself, hopefully I’ll find some time to look at the codebase in the winter months.
The server is listing 3 full and 7 incremental backups for this specific client. Zfs however has 60 datasets for it, totalling 32.8TB of used storage (urbackup statistics page says 7.23TB):
Ah I didn’t realize that script contains the deletion logic for the server. I’ve only played with the client side bash scripts, where I replaced most of the logic with python.
I’m going to try your suggestion on Monday. In the meantime, would it be worthwhile to polish my scripts and contribute them for integration? You’d have to pick up python3 and some py modules as dependencies, but my scripts are much easier to read and to maintain for all the different snapshot capable filesystems.
Not sure if this is the right spot, but trying to understand ZFS and Full file backups. It seems that if you run a full file backup (AFTER initial full backup and subsequent incrementals) with a ZFS back end, then ALL data is stored again on the new dataset, and not just the changes on top of the previous ro snapshot as in an incremental. Should not just the changes be stored? Kind of defeats the purpose of using ZFS…Is it recommended to not use Full file backups at all for ZFS (ie incremental forever?) If I want to keep a year’s worth of backups that leads to a huge! number of datasets…I would like to use a “traditional” setup with say 12 monthly backups and 30 dailies…without! storing all the full data 12 times…seems to me that should be possible with zfs?
