! Problems Now With urbackup_tmp_files

Server
#1

Damn it! Everything working fine yesterday morning. Computer only used for backups. Latest UrBackup v.2.4.13 server using a BTRFS filesystem for backups. I have plenty of space on single partition btrfs filesystem (~20% of 6.5TB). I don’t know how many file descriptors are available, but lsof | wc -l returns 17058. Rebooted. Now I get multiple lines in log file:

WARNING: Error opening temporary file. Retrying…

and last line is:

ERROR: Error deleting directory “/urbackup/urbackup_tmp_files”

So, I thought I would use CLI to delete and recreate the temporary directory. As root, if I use either “rm -r” or “rmdir”, I get a response of a Read-only filesystem.

Since the directory is not a subvolume, I assumed “btrfs subvolume delete” wouldn’t work, but… I tried it anyway:

“btrfs subvolume delete /urbackup/urbackup_tmp_files/”

but get:

ERROR: not a subvolume: /urbackup/urbackup_tmp_files/

btrfs fi df -h /urbackup/
Data, single: total=6.20TiB, used=5.08TiB
System, DUP: total=8.00MiB, used=704.00KiB
Metadata, DUP: total=46.00GiB, used=18.76GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

What can I do to recover my situation? Thank you!

#2

Also, dmesg gives me:

segfault at 55868cd07a9b ip 000055865dce9be7 sp 00007ffdd8d73840 error 4 in btrfs[55865dc9d000+a9000]

#3

Does mount | grep urbackup indicate that the filesystem is mounted read-only? In this case, you’d need to somehow recover your (corrupted?) BTRFS. BTRFS Wiki recommends this blog-post: http://marc.merlins.org/perso/btrfs/post_2014-03-19_Btrfs-Tips_-Btrfs-Scrub-and-Btrfs-Filesystem-Repair.html

#4

Yes, it is in readonly - Also, looking at dmesg I can see that it was forced to readonly. I’ve tried btrfs scrub and find that the partition has 4 uncorrectable errors. Then, trying btrfs check --repair ends in a segmentation fault.

I only have one drive at this time. Thanks for the link - I will check it out.

#5

I ended up mucking it up while trying to repair it. I just can’t seem to run urbackup to a btrfs partition longer than a year or so before the filesystem forces itself to readonly. I’ve never had an issue (with filesystem corruption) using ext4 for a file server. Not the same workload, but not idle very long either.

I ran mkfs.btrfs on the partition and am starting over. Again. I’ll feel better in a few days when all clients have had a chance to backup.