Windows Urbackup Server Out of Space (2TB in statistics, 8TB drive full)

Server
1

I’m running URBackup server 2.5.32 on a Windows 10 machine with the backup storage on an 8TB NTFS formatted drive.

I have 4 clients backing up (3 Windows, 1 linux), 3 are relatively small amounts of data, and 1 is pretty large. There is a mix of image and file backups. The image backups are insignificant in size (most less than 100GB, one is 300GB) and should total about 1TB. The File Backups are about 2-3TB of files.

The issue I’m running into is that even though the statistics show I’m only using 2TB of storage across all clients, the 8TB drive is full. I’ve tweaked the storage settings to keep a fairly small amount of backups, including only a single full file backup. Emergency cleanup keeps running, yet the storage space on the drive isn’t going down.

I’ve tried recalculating statistics, and it doesn’t change the value.

I ran into this before, and the only way I could “fix” it was to blow everything away and start over again, but unsurprisingly, the storage kept getting eaten up and now the drive is full again.

I’m not really sure what else to do at this point. Any ideas would be appreciated. Thank you!

2

I ran “remove_unknown.bat” and this cleaned up 3.5TB of space. I don’t know why this should be be necessary, but glad it helped.

However, now the statistics show 1.5TB used when the drive is 4TB full. I don’t think the statistics are working correctly.

3

If the filesystem is BTRFS - then you should use the tools specifically to that FS - including “balance”. I know you said Windows - but, could be a docker module.
ALso, try copying FS to alternative media to see if you get the same size. If no - then some sort of corruption of FS?

4

Thank you for the response. The file system is NTFS.

I ran a chkdsk which verified that the file system is OK (took a bit long because backups & NTFS compression were running at the same time):

Windows has scanned the file system and found no problems.
No further action is required.

   7630867 MB total disk space.
   5867512 MB in 3294602 files.
   2459628 KB in 388873 indexes.
         0 KB in bad sectors.
  18377859 KB in use by the system.
     65536 KB occupied by the log file.
1784839000 KB available on disk.

      4096 bytes in each allocation unit.
1953502207 total allocation units on disk.
 446209750 allocation units available on disk.
Total duration: 2.70 days (233517997 ms).

so no issues with the filesystem…

5

I suspect the statistics are correct.

This can happen because of NTFS cluster size and file system overhead.

NTFS stores files in fixed-size clusters (e.g., 4 KB, 8 KB, 64 KB).
Each file, no matter how small, occupies at least one full cluster.
If you have millions of tiny files (say, 1 KB each) and your cluster size is 64 KB, each file uses 64 KB on disk.

Example:
1,000,000 files×64 KB=64 GB
Scale that to millions more, and it adds up fast.

NTFS keeps metadata for every file in the MFT.
Millions of files = a huge MFT, which can consume tens or hundreds of GB.

NTFS journaling, security descriptors, and indexing add extra overhead.
Small files often store attributes in the MFT itself, but once that overflows, extra space is used.

You can check your cluster size with:
fsutil fsinfo ntfsinfo X:
where X: is your drive letter where you store your backups.

You can check if this is a significant cause for your problem by confirming actual size vs. allocated size.

There are likely many ways you can get this. An easy way is to use the free WinDirStat tool. It is very popular - search for it on the internet. When you open it you can select a single drive or let it look at all drives. when it finishes scanning, at the top on the drive letter you see Physical Size and Logical Size.

This is my Win11 laptop and shows that I have ~208GB of data taking up ~262GB of space.

image
image1920×1079 239 KB

My recommendation would be to consider reformatting with smaller cluster size.

This is destructive so you’d need to stop the urBackup service, copy data off first, reformat then copy it back.

You could also look to enable NTFS compression but I personally would not look at that option unless you are in a real pinch. It does nothing for already compressed files, slows down disk writes and some backup software doesn’t handle ntfs compress well - i have no idea if urbackup cares.

6

Thank you for your response. It has a 4KB sector size right now. It could be all the symlinks that are causing problems.

You can check if this is a significant cause for your problem by confirming actual size vs. allocated size.

I’ve tried the built-in Windows tool for this, it scans forever and shows more size in use than is available on the drive. Likely due to the symlinks.

I can try the WinDirStat tool… see if it figures it out better.

UrBackup guide actually recommends turning NTFS compression on for Windows servers. I did… it took a whole month to compress everything… helped a little lol.

Thanks, I’ll see how this goes.

Edit: Also, I would expect statistics to use Size on Disk… but maybe they don’t.

7

1 hour into the scan, it’s at 16.5TB physical size, 18.4TB logical size… on a 8TB drive. Yep.

Edit:
It finished scanning. 15.3 million files, 19.2TB Physical Size, 21.2TB logical size… so that’s not super helpful.

15 million files is only 60GB at 4KB sector size.