My appliance powered off after a prolonged power outage which outlasted the UPS. When it came back up, I’m getting the message in the title of this post and no backups are working. After a few hours I restarted thinking it might just be hung. I got the same message after it came back up and so I figured I’d wait it out. It has been over 24 hours now and still the same message and no backups.
Any ideas what I need to do to start troubleshooting and hopefully get backups running again?
Could you take a look/send /var/log/app.log and /var/log/raiddrive.log (assuming you use RAID)?
Raiddrive.log: https://drive.google.com/file/d/1YsudYpyPYBqGOGVJYF4Da2ghuan-jZns/view?usp=sharing
App.log: https://drive.google.com/file/d/1PCM-VsJYOPWtD6rsnZ_kLJHh98T7mqc4/view?usp=sharing
Thx! You might have to reset the cache completely.
What does /var/log/kern.log / dmesg say?
Cache corrupted, e.g.:
May 26 12:34:39 KingNAMSBackup kernel: [ 65.314493] BTRFS error (device dm-0): parent transid verify failed on 1034009296896 wanted 390013 found 389928
May 26 12:34:39 KingNAMSBackup kernel: [ 65.315184] BTRFS error (device dm-0): parent transid verify failed on 1034009296896 wanted 390013 found 389928
May 26 12:34:39 KingNAMSBackup kernel: [ 65.317345] BTRFS error (device dm-0): parent transid verify failed on 1034009296896 wanted 390013 found 389928
May 26 12:34:39 KingNAMSBackup kernel: [ 65.317414] BTRFS info (device dm-0): no csum found for inode 263 start 1695498240
The error even indicates that the cache device doesn’t honor FLUSH commands. If possible I’d be interested in what kind of hardware it is.
To recover best reset the whole thing and recover from backup storage like you did in Urgent! System Disk Failed - #5 by KingNAMS
If you don’t want cache corruption each time the power fails consider using different hardware as cache device.
I’m on a Dell PowerEdge R510 with a PERC H700 512BM Cache. So are you saying the PERC card may be bad? Or do you mean the SSDs that I have set up as the metacache? Those are just regular SATA SSDs plugged into ports 11 and 12 of the PERC card.
The cache device. NOT the RAID metadata cache devices.
And yeah, if it is an SSD, I’d be interested in the model.
Well I have attempted to reset the whole thing like you suggested but I’m not having luck there either. I followed the S3 guide and chose RAID recovery and entered the depcryption key. But then I got a message saying possibly the wrong key was used and now here is my status:
Yeah, looks like wrong encryption key. Plus you used one of your RAID metadata cache devices as cache device.
Is there a way to try again?
Yes, reset again… Make sure to not overwrite the other RAID metadata cache device, since it is the only mirror left…
Hey guys, I know this an old topic but one of the top results on Google. I had a similar issue, and nothing was working, even after resetting the cache. I was able to fix the issue by completely removing my virtual cache disk from my VM, and then logging in and removing offline storage devices. Once this was done, I added a completely fresh cache drive an voila - cache storage working once again, and my S3 backups were kept in-tact as well.
