Deduplication without BTRFS/ZFS

Feature requests
1

Firstly, I want to thank Uroni and anyone else involved with UrBackup development. I’ve been a UrBackup user for about 3 years now. UrBackup is a wonderful piece of software and replacing it has been really difficult - I’ve had to cobble something together and it’ll be a while before it works as well as UrBackup.

But - I feel it’s a necessity because I don’t feel I can trust BTRFS again. As soon as I try to do anything major with it, like add another disk, heavy I/O or upgrade the kernel - it has far too high a risk of breaking. I’ve already reinstalled the system a couple of times due to BTRFS issues and now a third time is just too much.

I was using BTRFS for deduplication, since UrBackup doesn’t support deduplication in software. I have previously tried using EXT4 and symlinks but it was just too slow to take regular backups. My understanding is that right now there is no deduplication in the software itself - it’s basically done at the filesystem level with snapshots in BTRFS or filesystem deduplication in ZFS which uses lots of memory?

Right now, I’ve switched over to Borg Backup. This does deduplication in software, so I can still use EXT4. But, it doesn’t have centralised management, a web interface, statistics, etc, so I’m having to build a lot of that myself.

I will keep an eye on UrBackup and if deduplication is ever supported properly in software, then I will jump back straight away.

2

Hi,

sorry for reviving this old post, but did you ever find a satisfying backup solution? I’ve tried five or six now, but none got all the features I’m looking for. Just did multiple hours of research and it seems like all solutions that actually support deduplication only allow for file level backups, not full images.

3

Hi. No - nothing open source or free. I now use a combination of things.

I’m still using borg for file backups from each individual server to a central server. I have it set up and managed by Ansible. It works really well. I use a combination of borg and borgmatic and then have a central script that runs, connects to the backup server and runs some borg commands to determine when each server last backed up and treat that as a success or failure flag.

I don’t really have the need to back up images manually. I use Proxmox on my servers, which has its own backup solution that I use for the actual server images and containers.