Deleted files are referenced in newer shadow copies

[Alexander]

Hello,

Had a few backup jobs running over the weekend from the same Server 2016 machine. All of them incremental file backups. The first six jobs/retries ended early with this or similar:

Info
	
2024-03-15 00:10
	Starting scheduled incremental file backup...
Info
	
2024-03-15 00:30
	Scanning for changed hard links on volume of "D:\"...
Fel
	
2024-03-15 00:30
	Error while getting files in folder "\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5\Profiler\PeterG.V6\AppData\Roaming\Mozilla\Firefox\Profiles\rsftfmm7.default\storage\default\https+++www.youtube.com\idb\1131144353LCo7g%sCD7a%tda2bdads.files". Windows errorcode: 87. Access to root directory is gone too. Shadow copy was probably deleted while indexing.
Fel
	
2024-03-15 00:30
	Indexing files failed, because of error
Fel
	
2024-03-15 00:30
	Constructing of filelist of "Radio1" failed: error - index error
Fel
	
2024-03-15 00:30

The last job managed to end but with issues, the log gave 25 of these errors for different files in the same firefox profile directory of a single user:

Fel
	
2024-03-17 19:53
	Error getting complete file "1BwqRFofRk1lurv1eXp4|D/Profiler/PeterG.V6/AppData/Roaming/Mozilla/Firefox/Profiles/rsftfmm7.default/storage/default/https+++www.youtube.com^partitionKey=%28https%2Cgoogle.com%29/idb/2805597024LCo7g%sCD7a%tfadbca1s.files" from Radio1. Errorcode: CANNOT_OPEN_FILE (3)

Looking at the parent folder, in this case D:\Profiler\PeterG.V6\AppData\Roaming\Mozilla\Firefox\Profiles\rsftfmm7.default\storage\default\https+++www.youtube.com^partitionKey=%28https%2Cgoogle.com%29\idb, it is indeed missing the mentioned file and the last modified date is 08-03-2024. This seems to be the case for all of the parent directories referenced.

vssadmin list shadows shows no stray copies left behind, vssadmin list shadowstorage shows 0 bytes is used for the volume in question.

Tried restarting the urbackup services on the client and server, then running a new incremental file backup, again it failed referencing a folder that does not exist, with a parent folder that was last modified three days ago.

Any ideas why this might be, and how to solve?

Best regards
Alexander

[ProfoundIT]

I don’t have a great answer, but could this have anything to do with file permissions in Windows? I don’t like backing up files that are a part of an active application while the backup is running. File locks maybe caused some sort of incorrect synchronization while a backup was running. Not sure, but that’s where my head goes.

[Alexander]

My spider sense is saying no, since the files do not exist any more on the client. I find it highly unlikely that the files giving us trouble are some kind of superhidden system files or similar, as the file names indicate that they are of the same type as files we can see in the same root folder.

I don’t know if this image is relevant or not, but on the left is the client folder in question that has not been touched since the March 8, on the right, an image of the resulting backup from March 22

What is interesting is that there are a number of folders that exist in the backup directory that are not present on the client, and the backup complains that it cannot find a number of files that are not present on the client, for instance:

Error getting complete file "1BwqRFofRk1lurv1eXp4|D/Profiler/PeterG.V6/AppData/Roaming/Mozilla/Firefox/Profiles/rsftfmm7.default/storage/default/https+++www.youtube.com/idb/1131144353LCo7g%sCD7a%tda2bdads.sqlite" from Radio1. Errorcode: CANNOT_OPEN_FILE (3)

This was an incremental backup, but not a resumed incremental backup. One thing that struck me was to test a full backup, I don’t know if this is a coincidence or not (the warning reported is most likely unrelated, the inability to remove a temporary backup folder from the server):

But I more and more suspect that there might be an issue related to persistence with incremental backups, even though it might need very special circumstances for it to show. I can’t find any other reasonable explanation right now at least.

Best regards
Alexander