Mount operation in progress. Backup storage will become available shortly

My appliance powered off after a prolonged power outage which outlasted the UPS. When it came back up, I’m getting the message in the title of this post and no backups are working. After a few hours I restarted thinking it might just be hung. I got the same message after it came back up and so I figured I’d wait it out. It has been over 24 hours now and still the same message and no backups.

Any ideas what I need to do to start troubleshooting and hopefully get backups running again?

Could you take a look/send /var/log/app.log and /var/log/raiddrive.log (assuming you use RAID)?

Raiddrive.log: https://drive.google.com/file/d/1YsudYpyPYBqGOGVJYF4Da2ghuan-jZns/view?usp=sharing

App.log: https://drive.google.com/file/d/1PCM-VsJYOPWtD6rsnZ_kLJHh98T7mqc4/view?usp=sharing

Thx! You might have to reset the cache completely.

What does /var/log/kern.log / dmesg say?

kern.log.txt (3.2 MB)

I’m not sure what dmesg means but here’s the kern.log.1 file

Cache corrupted, e.g.:

May 26 12:34:39 KingNAMSBackup kernel: [   65.314493] BTRFS error (device dm-0): parent transid verify failed on 1034009296896 wanted 390013 found 389928
May 26 12:34:39 KingNAMSBackup kernel: [   65.315184] BTRFS error (device dm-0): parent transid verify failed on 1034009296896 wanted 390013 found 389928
May 26 12:34:39 KingNAMSBackup kernel: [   65.317345] BTRFS error (device dm-0): parent transid verify failed on 1034009296896 wanted 390013 found 389928
May 26 12:34:39 KingNAMSBackup kernel: [   65.317414] BTRFS info (device dm-0): no csum found for inode 263 start 1695498240

The error even indicates that the cache device doesn’t honor FLUSH commands. If possible I’d be interested in what kind of hardware it is.

To recover best reset the whole thing and recover from backup storage like you did in Urgent! System Disk Failed - #5 by KingNAMS

If you don’t want cache corruption each time the power fails consider using different hardware as cache device.

I’m on a Dell PowerEdge R510 with a PERC H700 512BM Cache. So are you saying the PERC card may be bad? Or do you mean the SSDs that I have set up as the metacache? Those are just regular SATA SSDs plugged into ports 11 and 12 of the PERC card.

The cache device. NOT the RAID metadata cache devices.
And yeah, if it is an SSD, I’d be interested in the model.

Well I have attempted to reset the whole thing like you suggested but I’m not having luck there either. I followed the S3 guide and chose RAID recovery and entered the depcryption key. But then I got a message saying possibly the wrong key was used and now here is my status:


Yeah, looks like wrong encryption key. Plus you used one of your RAID metadata cache devices as cache device.

Is there a way to try again?

Yes, reset again… Make sure to not overwrite the other RAID metadata cache device, since it is the only mirror left…

How do I know which encryption key to use? There are 23 listed on my app.urbackup.com page