Linux device mapper snapshots filled up my disk space

client
1

Today I’ve had some spare time and felt adventurous and decided to try out the linux device mapper snapshots eventhough I couldn’t find any documentation about it. Neither how it works in urbackup nor in general. When looking for general documentation, I always only find information about lvm. Anyway.

After installing the urbackup client, it offered, to change the root device to a dm device (or something similar) to which I said yes. That worked so far, also after reboots. I don’t really now what exactly it did, and how I could undo those changes.

Then after some mainly unsuccessful tests and restarts I realized that I had no disk space left.
As it seems there where quite a lot of /.overlay_2fefd007-* files of 5GB size which filled up my complete disk space.
When the machine does a restart while doing a backup, those files just stay. Also when the backup fails the file stays (not always but often, more testing needed there).
The annoying part about is, you can’t just delete those files.
First you need to execute
chattr -i .overlay_2fefd007*
Which is easy enough, if you know it…

Also a lot of times the backup just gets stuck for hours in indexing, seemingly not doing anything (the overlay file was already created then) while when it works, the indexing only takes a few minutes.

I’ve managed to get a debug log from the backupclient when that happens. I’ve truncated it, it just continues like that for hours…
urb.log (19.4 KB)

My best guess is that something fails in /usr/local/share/urbackup/dm_create_snapshot and urbackup doesn-t properly handle that.
But without proper documentation about what is supposed to exactly happen and debug output from urbackup, it’s difficult to debug for me.
Anyway, it would be a cool features if it worked properly. I’d be willing to do some further testing, but would need some help there…

2

Yeah, looks like it hangs during the snapshot creation step. I guess you could narrow it down by looking at what’s currently called in that script?

To test one can run the snapshot script manually:

/usr/local/share/urbackup/dm_create_snapshot "/" "testsnap"
/usr/local/share/urbackup/dm_remove_snapshot "testsnap" "/mnt/urbackup_snaps/testsnap" "" "" "/"

(adjust ‘testsnap’ to a unique name per snapshot).

E.g. do a set +x on top to make it print the commands it executes.

Deleting the overlay files while a snapshot is active might corrupt the filesystem. That’s why they are protected as much as possible.

To remove, delete the initramfs hook, update initramfs, then reboot.

The only documentation of dm-snapshot (and dm-era) I found was the Linux one: Device-mapper snapshot support — The Linux Kernel documentation

3

I was always looking for them in /etc/initramfs/hooks. Didn’t know there are also hooks below /usr/share/initramfs-tools/hooks

That’s the only documentation I found myself. I’ve got to admit that I’m pretty impressed you managed to implement it using only that :+1:

So now to debugging. I-ve added “set -x” to the beginning of the script, and executed it → see attached file. Couldn-t quote it properly here.

urb.log (4.3 KB)

As you can see, the script failes executing urbackupclient_dmsnaptool. And from there it never exits. It just prints the error message and then hangs. Only way to exit is ctrl+c. So I guess that-s why the urbackupclient hangs for ever there. But I’ve got no idea how exactly to interpret the error message.

Using your above suggested dm_remove_snapshot command helped to get rid of the .overlay_2fefd007* file.

4

Looks like it hangs waiting for udev. Use dmsetup udevcookies to confirm.

Some problem with udev would also explain that /dev/mapper/root-98d1f8b1f435-a31725acca86421d-origin exists but isn’t a device mapper device…

I haven’t done any udev debugging previously. But perhaps something like running

udevadm monitor -e

after reboot and while it snapshots gives more information about the issue?

5

That works. IT replies

Cookie Semid Value Last semop time Last change time
0xd4dcc62 6 1 Thu Nov 17 20:08:26 2022 Thu Nov 17 20:08:26 2022

Actually urbackup automatically updated the client to 2.5.22
And now it seems to be working. Did you change something which could influence that?

Now I only get a warning:

Error opening file to lock: //.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_30e21d5a04d2bc39dfc99ecc04a6d23c18641ff28c7b2b2b. Operation not permitted (code: 1)

but the backup runs through anyway