Cleanup leftover overlay files and snapshots from linux client

Hi,
I am running a recent urbackup server on NAS in docker.
Client is ubuntu 22.04, urbackup is configured to use LVM snapshots (not dattobd),
and I tried (unsuccessfully) to do image backups yesterday.
Today, client has booted into a snapshot, and my /boot has no space left because of a large 5Gb /boot/.overlay* file.
=> I am missing the big picture what happened (a snapshot was created, so that the filesystem could be imaged, but due to interruption/failure it was not cleaned up so I am stuck in the snapshot ?)
=> I need to 1) revert grub back to booting from the plain LVM volume, 2) discard the snapshot and 3) delete the .overlay file. I hope I won’t have to do that manually, but what is the correct way ?
=> How to prevent that from happening in the future ? Not go for image backups at all ?
Thanks in advance,
yours,
Steffen

Hi, issue remains open. I’ll happily accept recommendation in what order I need to manually remove the snapshots in both / and /boot on the client.

The / snapshot is mounted: /dev/mapper/root-98d1f8b1f435 on / type xfs
Although I can’t find it in lvdisplay (see below). Or did I look the wrong way?
And the snapshot(s) appears under /dev/mapper/root* (see below)
and I see /.overlay_* files (see below).

The /boot snapshot is different, it appears not mounted: /dev/mapper/ssd-boot on /boot type ext4
It is not shown in lvdisplay (see below), and no snapshots appear under /dev/mapper/*boot*,
yet I do see /boot/.overlay_* files (see below).

Thanks in advance,
yours,
Steffen

root@happy:/# lvdisplay /dev/mapper/ssd-root 
  --- Logical volume ---
  LV Path                /dev/ssd/root
  LV Name                root
  VG Name                ssd
  LV UUID                1ckH2u-JPs5-wOD6-8p4O-gHfB-AJS5-HrzooX
  LV Write Access        read/write
  LV Creation host, time ubuntu, 2019-03-08 21:04:53 +0100
  LV Status              available
  # open                 2
  LV Size                36,00 GiB
  Current LE             9216
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

root@happy:/# ls -la /dev/mapper/root*
lrwxrwxrwx 1 root root 7 Jun 30 14:40 /dev/mapper/root-98d1f8b1f435 -> ../dm-8
lrwxrwxrwx 1 root root 8 Jun 30 14:44 /dev/mapper/root-98d1f8b1f435-a31725acca86421d-clone -> ../dm-20
lrwxrwxrwx 1 root root 8 Jun 30 14:44 /dev/mapper/root-98d1f8b1f435-a31725acca86421d-clone-era -> ../dm-17
lrwxrwxrwx 1 root root 8 Jun 30 14:44 /dev/mapper/root-98d1f8b1f435-a31725acca86421d-era-metadata -> ../dm-18
lrwxrwxrwx 1 root root 8 Jun 30 14:44 /dev/mapper/root-98d1f8b1f435-a31725acca86421d-era-metadata-access -> ../dm-19

root@happy:/# ls -la /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_*
-rw-r--r-- 1 root root          0 Jun 24 18:42 /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_4c90e6bd9b05f65a288e69575981efebf52d33041ea28427
-rw-r--r-- 1 root root          0 Jun 23 08:41 /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_cbe98c116d0387f513be218bb311231a31d265054ab00959
-rw-r--r-- 1 root root 5368709120 Jun 22 19:49 /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_cf47acbf27899af308e33be25fbfd54f929476a4ffe5e120
-rw-r--r-- 1 root root  104857600 Jun 22 22:22 /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_cf47acbf27899af308e33be25fbfd54f929476a4ffe5e120-wsnap
-rw-r--r-- 1 root root          0 Jun 30 14:44 /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_eca2bad1b692ede4f5585f4c1c8a4b04f6896ded872e192c

And output for /boot:

root@happy:/# lvdisplay /dev/mapper/ssd-boot 
  --- Logical volume ---
  LV Path                /dev/ssd/boot
  LV Name                boot
  VG Name                ssd
  LV UUID                8KrJbF-e4KI-1pOL-HYvu-Z4ki-Uqqr-0KkCNl
  LV Write Access        read/write
  LV Creation host, time ubuntu, 2019-03-08 21:09:56 +0100
  LV Status              available
  # open                 1
  LV Size                5,00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

root@happy:/# ls -l /dev/mapper/*boot*
lrwxrwxrwx 1 root root 7 Jun 30 14:40 /dev/mapper/ssd-boot -> ../dm-3

root@happy:/# ls -la /boot/.overlay_*
-rw-r--r-- 1 root root   89726976 Jun 23 20:07 /boot/.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_6ea1e9df3cb55e34bc282f317e9d39e29f9b14ed6ed66ec9
-rw-r--r-- 1 root root 4980699136 Jun 22 19:46 /boot/.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_cf2796d29caa4b25473a72107d2d2391b159a1d1c8b88892
-rw-r--r-- 1 root root          0 Jun 24 18:54 /boot/.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_f1b1640af29319834158c96a97c88a80e77d3c0dbb475319