I’m looking at setting up urBackup on my home server as a backup solution for our household. Our mixture of OS clients (osx/windows/linux) currently have just over 1.5TB of data.
My server:
Has 2 x 2TB drives available for backup
8GB RAM
A free internal USB slot.
Is running Ubuntu 16.04 on a separate 250GB spinning disk.
(There are also 2 x 1TB hard disks in an mdadm raid1 array that contain media files.)
It would seem that 2TB is sufficient capacity for the moment so I should be able to to combine the two 2TB disks into another raid array – but how? AFAIK mdadm has no compression or dedup features, though it’s been perfectly stable for me. I presume I need to set up a proper ZFS pool using both disks to use that – but my impression is that 2 disks is not worthwhile with ZFS and can spell unrecoverable disaster if one fails. Btrfs I know little about but presume this needs similar setup to ZFS.
I’m more interested in redundancy than performance. Before discovering urBackup I was planning on backing up to one disk and simply periodically cloning that disk to the other (and cloning to an off site service too).
So the question is, what would you do with this setup? How should I make best use of those two empty 2TB disks?
You can put the two 2TB drives into a ZFS mirror with something like zpool create BACKUPPOOL mirror /dev/sdc /dev/sdd. That will give you all of the benefits of ZFS. You can enable compression, but ZFS deduplication is not recommended. Rely on urbackup to handle file level deduplication.
With only 8GB of RAM, you will need to adjust the ZFS arc size to be smaller. I run ZFS on a system with 8GB of RAM, and it would occasionally freeze when I left the default arc size. Limiting arc to only 3GB has prevented any freezes. To do this, create /etc/modprobe.d/zfs.conf and put the line options zfs zfs_arc_max=3221225472 into it.
BTRFS might be a good option, but I have not played with it in a few years.
BTRFS works great with UrBackup, just make sure you put the backup path into /etc/urbackup/backupfolder so UrBackup knows it can use btrfs snapshots. Then set the image backup type to raw/cow. Each image/file backup is then performed into a snapshot of the previous backup of the same type. If you only change one part of one file then that is the only part that is COW’d and you save a lot of disk space without needing to use soft/hard links. Each incremental snapshot is essentially a full backup all by itself.
If you use raid1, I worry that your 2TB of space will not be enough for 1.5TB of data. Once you account for multiple image and file backups I fear you will run out of space quickly.
Thanks. That ‘only 8GB of RAM’ for zfs is one of the things that puts me off, powerful as zfs is, it just seems like overkill for my needs.
I’m going to give it a shot with BTRFS, thanks for heads up about the space. I’ll either need to be more selective about what I back up or not use raid1.
ZFS doesn’t “require” 8GB of RAM. ZFS (like most other file systems) benefits from more free RAM and will (through ARC) attempt to maximize usage of RAM. ZFS can be tuned to require way less (trading performance). ZFS can run on as little as 1GB (although you’ll have almost no caching).
ZFS is nice because it verifies your data continuously and trades your performance for reliability, hence the use of memory and SSD to speed things up. If the reliability of your data is less important to you, you can easily use XFS or EXT4.
BTRFS is (still) very unstable and has the same ‘issues’ in regards performance as ZFS, I wouldn’t recommend it if your data is important to you, I’d suggest a more stable system. It also is ‘heavier’ than ZFS as far as code, has less support across OS and lacks important features.
Just as a housenumber… i’m running a backup server at work with urbackup with 4x3tb WD Red drives in a RAIDZ2 configuration (RAID6 style) and backing up about 50 machines to it… Storage used is about 4TB…
afaik RAID1 is considered stable for btrfs and for zfs so it does not really matter which one you choose.
BTRFS has the advantage that urbackup can use the snapshot mechanism wich makes backups way faster.
ZFS is considered the more stable FS but urbackup can not use the zfs snapshots. therefore every new backup has to use hardlinks which takes a bit longer…
Both FS have a good LZO compression on board wich usually results in a good and fast compression.
In my particular usecase i end up with about a 1.5 ratio… could save a little bit more with dedup (1.2) but thats simply not worth it
i had to use ZFS because the parity raid (RAID5/6) implemenation is considered unsafe for BTRFS and i personally always choose raid 5 or 6 for pure datastorage over raid10.
finally… i think you’re mixing up things here…
MDADM is not a filesystem. its just a “software raid controller” (so to speak).
it’s only used to create a raid and ontop of that a filesystem is created.
ZFS and BTRFS are both “mixed” filesystem wich have their own raid functionality implemented so you don’t need to use mdadm or any raid controller at all. for ZFS it is even recommended to use the raw devices.
Yes, tarakesh, I am mixing things up! I’d always assumed ZFS/BTRFS dealt with RAID themselves – but I’ve been using LXC a little recently and that uses a ZFS pool for storage. That made me realize ZFS pools don’t have to be device based…add in the suggestion that Linux isn’t the best for ZFS, BSD is better etc and I started to wonder if mdadm isn’t a more stable option and could/should it be used in conjunction.
It seems a lot of these issues are historical though, from everyone’s comments I’m happy to go ahead with zfs/btrfs alone. I still think ZFS is overkill for my home backup though and I have great worries about what would happen if one of the two disks died when using ZFS, or the complexity of merely increasing the size of the disks.
Well… ZFS does not have to be device based… it’s just best practice to do so because then ZFS knows the device geometries and can make best use of the drive caches.
It’s also correct that ZFS is way more stable on BSD than on linux… Just because it originates on BSD.
for your scenario i would personally use zfs mirror with compression
In that way you are getting the most out of your backup storage…
why? because i think zfs is more stable than btrfs. even considered the fact that zfs is relativly new on linux…
and in my oppinion zfs has a easyer to unserstand cli…
zpool create mirror MIRRORNAME /dev/disk/by-id/diskname1 /dev/disk/by-id/diskname2 -m mountpount
zfs set compression=on MIRRORNAME
zfs set atime=off MIRRORNAME
done
you do not have to fiddle with fstab or mount options at all and always use the /dev/disk/by-id because then you don’t have to mock around with changing disk names (sda sdb …) and always use the full disk… not partitions
if a disk dies on zfs: zpool replace MIRRORNAME /dev/disk/by-id/newdisk
wait for resync. done
i would not say that mdadm is a more stable solution… in my eyes they are more or less equal with a little advantage on the zfs/btrfs side… a resync on a mdadm system is always done on the full disk sizes because mdadm has no clue of any filesystem at all. btrfs/zfs can do a much quicker resync because they know which blocks are in use and which not.
If you go for a raid1 it does not matter at all wich system (mdadm/btrfs/zfs) you use… all data is mirrored in realtime on both disks. btrfs/zfs are just faster on the resync.
Before making THE decision I would recommend trying out raid1 resyncing in all scenarios on your own skin. Just create LVM volumes and test the thing. What good a raid1 would be if you cannot support it?
Yes, btrfs and zfs will sync faster than mdadm. The latter syncs empty space as well.
If you are not willing to use compression and deduplication on filesystem level I would still recommend mdadm. If anything goes bad and you loose your root file system, you can recover this using any linux usb distro. With ZFS it might not be the case. Also on mdadm you can enlarge the array by replacing disks, syncing (repeat) and than growing the array. (With ZFS this will not fly. this is not true actually)
And one more thing - place your root file system on the raid1 as well. Why leave it on a separate disk?
the probability of failure increases with number of drives
How could you loose backups due to having rootfs on the same array on a different partition or different LVM volume? User error is possible but the risk is the same regardless of configuration
IMHO it’s generally ‘easier’ if your backups reside on a different disk, because you can always reinstall your OS or even change the OS and move your disks along. I’ve moved one pool from OpenSolaris to Nexenta to FreeNAS over the last decade.
As far as rebuilds, ZFS has the benefit it will detect issues with your drive before MDADM even has a clue. I’m a bit jaded because I support an environment with currently well over 100 hard drives (200TB main, 180TB backup, 120TB off-site backup). Failure rate is ~2-3%/year and ZFS detects at least one checksum error every single month per array, somewhere in the chain a bit falls over undetected, no SMART errors but the checksums don’t match, these are drives varying from 3 months-7 years old and sometimes a series of checksum errors means the drives are failing, without any other indication.
The probability of data loss over time is actually pretty high. With a ‘simple’ MDADM RAID5 over a set of disks this can accumulate as high as 25% over the next 10 years. There are calculators that will tell you the probability over the next n years given a specific set of disks but I wouldn’t trust RAID5/RAIDZ, either go a series of RAID1 with spares or RAID6/RAIDZ2.
Increasing the size of your pool is relatively easy. You can either add more devices, or you can replace the hard drives with larger ones. This can be done on the fly without downtime.
