Urbackupclient wrote 2TB of logfiles until disk was full

Urbackupclientctl version used: 2.4.9 on Debian Stable.
I’ve got an early call today, since a server was responding anymore.
Analyzing it I’ve found a syslog of almost 2TB size.
As it seems urbackup was running amok.

It was writing following line endlessly, 10 000 of times, into the logfile. All with the same timestamp and referencing the same file.

05:12:01 intranet urbackupclientbackend[658]: ERROR: Cannot stat “/mnt/urbackup_snaps/f10cc91c1fad12c0b83b32e3ce3bb5d3b8f9163b681cc3c4/usr/src/linux-headers-4.19.0-9-amd64/include/config/ldisc”: No such file or directory (2)

I didn’t check when exactly timestamps would change, since I needed to get the server running as quickly as possible again.
Unfortunately I also didn’t have any possibilities to quickly copy 2TB somewhere, so the log file is lost.

From time to time there were following two messages in the log:

05:12:01 intranet kernel: [44099.776054] EXT4-fs warning (device dm-17): htree_dirblock_to_tree:995: inode #112596575: lblock 15: comm file indexing: error -5 reading directory block

05:12:01 intranet lvm[305]: No longer monitoring snapshot intranet–vg-urbackup_snap_f10cc91c1fad12c0b83b32e3ce3bb5d3b8f9163b681cc3c4.

There were also like 12 old lv snapshots from urbackup around which i’ve removed. I’ve got no idea what other information I could give you.

I’ll look into how it could go into such an infinite loop on error…

The catastrophic outcome could probably be prevented by limiting syslog size (/etc/logrotate.d/rsyslog size 10M), preventing the client to log into syslog at all (add StandardOutput=null to service file) or rate limiting journald more aggressively.

You should definitely run a e2fsck on that volume…