Windows: race between parallel file-metadata application and hashdata writes — "Error truncating hashdata file (code: 32)", "wrong size" metadata, misleading "FATAL: disk problems"

cina · August 28, 2026, 4:09pm

Environment

UrBackup Server 2.5.37 on Windows Server 2025 Datacenter (NTFS, 64 KB clusters, tiered Storage Spaces), storage healthy (chkdsk /scan clean, ~10 TiB free)
Windows clients 2.5.32 (local LAN), Linux clients 2.5.31
hash_threads=1, client_hash_threads=1, all “parallel file*” advanced/beta settings at their defaults (1), deduplication disabled
Affected: file backups of Windows clients with large system volumes (C:\ incl. WinSxS, WindowsApps, EdgeWebView/EdgeCore components)

Symptoms (server urbackup.log)

WARNING: File "\\?\F:\BACKUP\<client>\<backup>\.hashes\C\...\identity_helper.Sparse.Stable.msixbundle" has wrong size. Should=48 is=672. Error writing metadata to file. -1
ERROR: Error truncating hashdata file -2. The process cannot access the file because it is being used by another process. (code: 32)
ERROR: Error writing file metadata -1
ERROR: Error writing file metadata to file "F:\BACKUP\<client>\<backup>\.hashes\C\Windows\WinSxS\...\app.appx"
WARNING: Error reading current metadata
ERROR: Fatal error during backup. Backup not completed
ERROR: FATAL: Backup failed because of disk problems (see previous messages)

It is repeated amongst various windows clients and various files fail repeatedly within one client (mostly EdgeCore/EdgeWebView/WindowsApps/WinSxS msix/appx files), but any hardlinked file can be hit.
No disk problem exists — the FATAL message is misleading.

Analysis (current dev branch sources)
Two threads in urbackup_srv access the same .hashes\<path> metadata file concurrently, with incompatible Windows share modes:

FileMetadataDownloadThread::applyMetadata / FileMetadataApplyThread (“fb meta apply”), started from FileBackup::startFileMetadataDownloadThread when the client announces FILE_META>0 (Windows clients hardcode FILE_META=1 in ClientServiceCMD.cpp), opens the metadata file with MODE_RW. In file_win.cpp, File::Open uses dwShareMode=FILE_SHARE_READ for MODE_RW (FILE_SHARE_WRITE/DELETE are only added for DEVICE/DELETE modes).
The main backup path (BackupServerHash, server_hash.cpp, the “successfully linked file” branch) copies hashdata from the previous version’s hashpath into the same file and then calls os_file_truncate (os_functions_win.cpp), which opens via _wsopen_s(..., _O_RDWR | _O_CREAT, _SH_DENYNO, ...). The requested read/write access conflicts with the still-open MODE_RW handle that only grants FILE_SHARE_READ → ERROR_SHARING_VIOLATION (32) → “Error truncating hashdata file -2”.
The subsequent write_file_metadata(out_fn, ...) (file_metadata.cpp, opens MODE_RW_CREATE) fails the same way (“Error writing file metadata to file …” + “-1”). If the open succeeds while the other thread has already appended metadata, the size check fails → “has wrong size. Should=48 is=672” (48 = expected hashdata size, 672 = hashdata + appended metadata). “Error reading current metadata” is the same file read mid-write.

Synchronization gap: MaxFileId (FileBackup.h) gates the metadata application thread on isFinished(metadata_id-1), but setMaxPreProcessed(line) is called right after the entry is queued in IncrFileBackup.cpp (after addToQueueFull), i.e. before the download/hash thread has actually written and closed the hashdata/metadata file for that entry. So the apply thread can open the file while the main/download path is still writing it.

Trigger / probability
Probability grows with the number of hardlink-metadata operations per backup: first backups after a restore/storage migration (change tracking reset → everything reprocessed), full backups, and first backups after long gaps. In our case it appeared right after migrating the backup folder to a new volume and ran for days until a retry succeeded.

Impact
Backups abort as FATAL; failed backup dirs are removed; eventually a retry succeeds, but large clients can lose file backups for days.

Suggested fixes

Serialize per-file access: mark entries “preprocessed” only after the hashdata/metadata file write is fully finished and closed (or gate the apply thread on actual write completion).
Alternatively open metadata files on Windows with FILE_SHARE_READ|FILE_SHARE_WRITE and make os_file_truncate use a compatible share mode.
Do not report metadata write failures as “FATAL: Backup failed because of disk problems”.

Related existing threads (checked before posting — this is not a duplicate of them)

T15543 – “Error writing metadata to file. -1 _ File has wrong size. Should=8 is=116” (SOLVED): same symptom family on a Linux server/client. Root cause there: non-default beta “number of parallel works to compute fingerprint” settings > 1; the fix was resetting them to 1. Our server runs all parallelism settings at their defaults (1), so that fix does not apply here, and the analysis above points to a different concurrency gap (apply thread vs. truncate/metadata write, not parallel hashing).
T15932 – “Error writing metadata to file. -1”: Windows client, huge first backup onto new storage; same wrong-size/FATAL pattern, eventually resolved by retry without a root cause. Same trigger class as ours.
T4719 – “Segfault on out of disk space”: the only previous report containing “Error truncating hashdata file” we could find — but with code 28 (no space left on device). Our variant is code 32 (ERROR_SHARING_VIOLATION).
T12494 – “Backup fails with disk errors but I can’t find any disk error”: about the misleading FATAL message itself.

Disclosure

Transparency note: the source-code analysis in this report was AI-assisted (reading the urbackup_backend sources), and the report itself was drafted with AI assistance, under human supervision and review. All log excerpts, versions and environment details above are from our own systems and were verified by us.