Backup Statistics not adding up

I’m continuing my quest to troubleshoot a particular client that is showing an excessive amount of used space, and I’ve found that the statistics calculation doesn’t seem to add up.

On this particular host, statistics are showing that it’s using 887 GB of space:


However, when I go to Backups → Select Client, and manually add up the sizes for the listed backups, I only get ~225GB. Where did the other ~650GB come from?

Furthermore, notice the most recent incremental backup at the top. It’s 165GB. However, that client is only using ~8GB of space. So how can the incremental backup be approximately 20x the actual used space on the client?

# df -h
Filesystem                       Size  Used Avail Use% Mounted on
devtmpfs                         960M     0  960M   0% /dev
tmpfs                            980M     0  980M   0% /dev/shm
tmpfs                            980M  8.6M  971M   1% /run
tmpfs                            980M     0  980M   0% /sys/fs/cgroup
/dev/mapper/rl_linux--new-root    17G  7.2G  9.9G  42% /
/dev/sda1                       1014M  380M  635M  38% /boot
tmpfs                            196M     0  196M   0% /run/user/606401104

Looking at the logs for this backup, it shows that only 785.499MB were transferred:


I’ve already run remove-unknown successfully (see previous post), and it did remove quite a few files, but I’m still seeing this issue.

Any suggestions for where to look?

Try to click “Recalculate statistics” in the statistics page.

I can’t believe that I never saw that button! However, it doesn’t seem to be doing anything. Maybe there’s something happening in the back-end that I’m not seeing, but it spins (in the top right corner of the web UI) for just a second, and then it brings me back to the same place with the same numbers.

When I check in the systemctl logs, I don’t see anything new, just my last Login successful message.

Have you checked the real disk store to see how much its storing. I understand one of those is the actual size, but it uses deduplication as well. That said it means none of the numbers should be more than the disk. Are you doing file AND disk backup as that will at least double the size at a minimum.

The NFS mount is a ZFS dataset on the NAS. The ZFS dataset currently shows a total of 2.65TiB used space, while UrBackup statistics report 4.1TB.

I’m only doing file and image on a single clients (you can see that client in the list at the top). Otherwise, I’m doing image backup only on Windows Clients and Servers, and file backup only on Linux servers.

This makes sense as there will be deduplication going on, so generally you should always be storing less on the real disk than whats shown in the statistics. So if 2 systems at 1Tb each were backed up, but the 2 systems were identical, your statics will show 2Tb of store but the disk will likely only have 1Tb of real data.

I’m not sure that deduplication can explain what I’m seeing.

For example, the total used space per statistics doesn’t match the total of the space for the backups, and the space on the backups doesn’t match the actual log file of the backups. (See my last picture in my first post).

Also, the host that shows the most used space (2.09TB in the screenshot). That client runs a different OS to all other backed up systems, and has loads of software that’s not duplicated across any other client. I would have a hard time believing that it’s using anywhere close to ~80% of the total storage given all the other data that’s being backed up.

If you use a space analyser like space sniffer you will probably see files named directory_pool taking up insane amounts of space for some clients im not sure why this is the case , see picture attached. so Urbackup shows that the clients is using 700GB but in reality its 1.8TB

Well here is the explanation

It’s been a few months, but I wanted to revisit this.

To start off with, I’m still at a loss as to what’s going on with the backup statistics. Even after going through and deleting all of the backups for this client, running remove-unknown, and recalculating statistics, there was still multiple hundreds of GB of storage space being reported by UrBackup. To verify, I confirmed that the backup folder was completely empty, and the filesystem host reported 0 bytes used by du.

It makes me wonder if UrBackup doesn’t properly “clear” its space allocation for deduplication, but I’m not quite sophisticated enough to dive in and figure that out.

Also, in other news, I’m pleased to report that I figured out why this particular client was actually using so much space compared to others (while the numbers themselves are wonky, they are relatively correct). A glitch in our ansible config meant that log rotation wasn’t working correctly on any of our internal web clients. This web client was particularly well used (multiple display TVs throughout the organization access its pages multiple times per day), which was blowing up its request logs. With log rotation properly configured now, this client isn’t showing any extra backup size.

It doesn’t really answer this question, but it at least de-prioritizes it for me a bit.