Urbackupsrv crashes on Ubuntu 20.04

Hello,
yesterday I upgraed my Ubuntu 18.04.4 to Ubuntu 20.04 with urbackup installed from the ppa.
I noticed that urbackupsrv is not running and started analysing. Since there is no 20.04 build on the PPA, I uninstalled urbackup and reinstalled it from the OpenSuse build service.

Running it manually starts well, but suddenly crashes with a Segmentation fault:

2020-04-29 16:46:42: Starting HTTP-Server on port 55414
2020-04-29 16:46:42: HTTP: Server started up successfully!
2020-04-29 16:46:42: SQLite: recovered 432 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 12904 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 18758 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 30273 frames from WAL file /var/urbackup/backup_server_links.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 432 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 12904 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2020-04-29 16:46:42: SQLite: recovered 30273 frames from WAL file /var/urbackup/backup_server_links.db-wal code: 283
2020-04-29 16:46:43: SQLite: recovered 18758 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2020-04-29 16:46:43: Started UrBackup…
2020-04-29 16:46:43: Removing temporary files…
2020-04-29 16:46:43: Recreating temporary folder…
TEST FAILED: guestmount is missing (libguestfs-tools)
2020-04-29 16:46:43: Image mounting disabled: TEST FAILED: guestmount is missing (libguestfs-tools)
Testing for btrfs…
TEST FAILED: Creating test btrfs subvolume failed
Testing for zfs…
TEST FAILED: Dataset is not set via /etc/urbackup/dataset
2020-04-29 16:46:43: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.
2020-04-29 16:46:43: Reflink ioctl failed. errno=95
2020-04-29 16:46:43: Broadcasting on ipv4 interface em1 addr 192.168.2.100
2020-04-29 16:46:43: Broadcasting on ipv6 interface em1 addr fe80::3e4a:92ff:fe74:3124
2020-04-29 16:46:43: InternetService: Server started up successfully!
2020-04-29 16:46:43: UrBackup Server start up complete.
2020-04-29 16:46:43: Server started up successfully!
2020-04-29 16:46:43: Looking for old Sessions… 0 sessions
2020-04-29 16:46:43: Downloading version file…
2020-04-29 16:46:44: Downloading version file…
2020-04-29 16:46:44: Downloading server version info…
2020-04-29 16:46:44: Downloading dataplan database…
2020-04-29 16:47:14: New Backupclient: CLIENT1
2020-04-29 16:47:14: New Backupclient: CLIENT2
2020-04-29 16:47:14: New Backupclient: CLIENT3
2020-04-29 16:47:14: New Backupclient: CLIENT4
Segmentation fault (core dumped)

Any idea on how to debug this problem? Thanks

Edit: In your Wiki I found an instruction on how to provide debug-information with gdb. So here is the output: https://www.codepile.net/pile/JL4bwvmg

1 Like

Could you install the debug symbols and rerun this? ( https://download.opensuse.org/repositories/home:/uroni/xUbuntu_20.04/amd64/urbackup-server-dbgsym_2.4.12.0-1_amd64.ddeb )

Of course, here you go: https://www.codepile.net/pile/pwGrMnVb

Hmm, can’t reproduce the problem and it seems to be easily reproducible… (just connect a Internet client)

So two ideas: Perhaps it is indeed your CPU (output of cat /proc/cpuinfo), or your ubuntu has a stale crypto++ lib (apt remove libcrypto++6 then install urbackup-server again…?)

Here is my /proc/cpuinfo https://www.codepile.net/pile/jWPgERM5
I would be surprised if its because of the cpu, with 16.04 and 18.04 I had no problems. I was running the same version of urbackup in 18.04 already before upgrading to 20.04

Reinstalling libcrypto++6 and urbackup-server didn’t help, same segfault.

I disabled my homes firewall rules to deny any access from the internet clients, urbackup is running now . including sucessfully doing backups in my home network. So just no backups of my internet clients for now…

I just had the same issue on upgrading from 18.04 to 20.04. Used to be working without any problems until the upgrade. Now, as soon as an Internet client connects, seg-fault and core-dump. Unfortunately, all my backup clients except one are Internet clients.

As a workaround, what works well, is running the urbackupsrv in the official Docker container. Internet clients can still connect in this case.

Solved installing previous version of libcrypto++6

wget http://ftp.br.debian.org/debian/pool/main/libc/libcrypto++/libcrypto++6_5.6.4-8_amd64.deb
sudo dpkg -i libcrypto++6_5.6.4-8_amd64.deb

I can confirm ivan’s workaround works!
Sadly with any apt upgrade or install it will break again, so it’s really just a short term solution to finally do some backups of the online machines.

Is the same occurring with the freshly build 2.4.13? (now also in the ppa)

Yes, firstly I have tried disabling the encrypted transfer in the internet settings, upgraded to 2.4.13 and it fails too.

I also tested yesterday with the very new 2.4.13 and had the same issue

Is there an update to this?

I’ve installed Ubuntu 20.04 on my home server and urbackup is crashing when an internet client connects.

I’ve ran the opensuse version for the debug symbols and got the following:

2020-08-09 15:42:39: InternetService: No available slots… starting new Worker
[New Thread 0x7fffb27fc700 (LWP 764074)]
–Type for more, q to quit, c to continue without paging–bt
Thread 27 “InternetService” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb27fc700 (LWP 764074)]
__memmove_sse2_unaligned_erms () at …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:383
383 …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.

Output of cpuinfo

vendor_id : AuthenticAMD
cpu family : 16
model : 6
model name : AMD Turion™ II Neo N40L Dual-Core Processor
stepping : 3
microcode : 0x10000c8
cpu MHz : 800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips : 2994.99
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

Seems a pattern that it happens with old AMD Athlon processors?

Also as ivan said, the problem seems to be in in the crypto++ package, which is not part of urbackup server.

In my case it’s an Intel(R) Core(TM) i7-4790 CPU.
It quite clearly is related to the packages in 20.04, since running the official docker image works fine (that’s my current workaround).

1 Like

Dual xeon’s here with same issue once I upgraded to Ubuntu server 20.04 from 18.04. Definitely an issue with the libcrypto++6 upgrade from 5.6.4-8 to 5.6.4-9.

As said, I cannot reproduce the issue when I try it e.g. on a DigitalOcean Ubuntu 20.04 instance.

Ran

wget http://ftp.br.debian.org/debian/pool/main/libc/libcrypto++/libcrypto++6_5.6.4-8_amd64.deb
sudo dpkg -i libcrypto++6_5.6.4-8_amd64.deb

like @ivan recommended. Then ran

sudo apt-mark hold libcrypto++6

Seemed to resolve issue. Have multiple backup servers running Ubuntu 20.04, and all of them couldn’t run urbackupsrv. Would crash with Segmentation Fault. Can do other updates after holding the package.

And have Intel and AMD and Server and Desktop processors in use at the moment.

I tried the package libcrypto++6_5.6.4-9 (same version as in Ubuntu) from debian/sid repo and it also solves the issue. So it looks like ubuntu-package-related (not library-version related)

wget http://ftp.ca.debian.org/debian/pool/main/libc/libcrypto++/libcrypto++6_5.6.4-9_amd64.deb
sudo dpkg -i libcrypto++6_5.6.4-9_amd64.deb
sudo apt-mark hold libcrypto++6

1 Like

Could someone having the problem file a bug? https://bugs.launchpad.net/ubuntu/+source/libcrypto++/+filebug