Segfault on server 2.1.20

Server
1

Hi

I’ve recently upgradet to 2.1.20 and I’m having some trouble with the server crashing every other day.

From dmesg:

[15724.221470] fbackup load[5312]: segfault at e0f040 ip 0000000000e0f040 sp 00007f13e87561f8 error 15
[15732.779685] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[15732.791345] JFS: nTxBlock = 8192, nTxLock = 65536
[15732.859249] ntfs: driver 2.1.32 [Flags: R/O MODULE].
[15732.948994] QNX4 filesystem 0.2.3 registered.
[15738.615897] audit: type=1400 audit(1512048723.084:17): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/lxd/lxd-bridge-proxy" pid=6432 comm="apparmor_parser"
[15744.300122] audit: type=1400 audit(1512048728.768:18): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/bin/lxc-start" pid=6740 comm="apparmor_parser"
[15744.364617] audit: type=1400 audit(1512048728.832:19): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/lxd/lxd-bridge-proxy" pid=6743 comm="apparmor_parser"
[15744.368442] audit: type=1400 audit(1512048728.836:20): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/libvirtd" pid=6747 comm="apparmor_parser"
[15744.387349] audit: type=1400 audit(1512048728.856:21): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/tcpdump" pid=6749 comm="apparmor_parser"
[15744.476515] audit: type=1400 audit(1512048728.944:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/sbin/dhclient" pid=6738 comm="apparmor_parser"
[15744.476809] audit: type=1400 audit(1512048728.944:23): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=6738 comm="apparmor_parser"
[15744.477076] audit: type=1400 audit(1512048728.944:24): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=6738 comm="apparmor_parser"
[15744.477333] audit: type=1400 audit(1512048728.944:25): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=6738 comm="apparmor_parser"
[15744.479699] audit: type=1400 audit(1512048728.948:26): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=6746 comm="apparmor_parser"
[15744.499168] audit: type=1400 audit(1512048728.968:27): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/libvirt/virt-aa-helper" pid=6742 comm="apparmor_parser"
[15751.109908] audit_printk_skb: 21 callbacks suppressed
[15751.109911] audit: type=1400 audit(1512048735.576:35): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/libvirtd" pid=7337 comm="apparmor_parser"
[15751.286375] audit: type=1400 audit(1512048735.752:36): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/libvirt/virt-aa-helper" pid=7340 comm="apparmor_parser"
[15778.184156] InternetService[15132]: segfault at c91040 ip 0000000000c91040 sp 00007f35d67fbd78 error 15
[73456.467428] cgroup: new mount options do not match the existing superblock, will be ignored

From urbackup.log after crash (loglevel debug):

2017-12-05 09:23:12: Updating file statistics: 91%
2017-12-05 09:23:12: File processing speed: 500000 files/s
2017-12-05 09:23:12: Updating file statistics: 94%
2017-12-05 09:23:12: File processing speed: 1e+06 files/s
2017-12-05 09:23:12: Updating file statistics: 98%
2017-12-05 09:23:12: Done updating statistics.
2017-12-05 09:23:12: Downloading version file...
2017-12-05 09:23:12: GT: Loaded file "[client file]"
2017-12-05 09:23:12: Loading file "[client file]"
2017-12-05 09:23:12: PT: Hashing file "[client file]"
2017-12-05 09:23:13: GT: File "[client file]" not found via hash. Loading file...
2017-12-05 09:23:13: GT: Linked file "[client file]"
2017-12-05 09:23:13: Downloading version file...
2017-12-05 09:23:13: HT: Copying file: "/mnt/backup1/urbackup/[client file]"
2017-12-05 09:23:13: GT: Loaded file "[client file]"
2017-12-05 09:23:13: Loading file "[client file]"
2017-12-05 09:23:13: PT: Hashing file "[client file]"
2017-12-05 09:23:13: HT: Copying file: "/mnt/backup1/urbackup/[client file]"
2017-12-05 09:23:13: Downloading version file...
2017-12-05 09:23:13: GT: Loaded file "[client file]"
2017-12-05 09:23:13: Loading file "[client file]"
2017-12-05 09:23:13: PT: Hashing file "[client file]"
2017-12-05 09:23:14: HT: Copying file: "/mnt/backup1/urbackup/[client file]"
2017-12-05 09:23:14: GT: Linked file "[client file]"
2017-12-05 09:23:14: GT: Linked file "[client file]"
2017-12-05 09:23:14: Downloading server version info...
2017-12-05 09:23:14: Downloading dataplan database...

I’m running Ubuntu 16.04, Linux 4.4.0-101-generic

Please advise.

2

I don’t think anything was changed in that are. Can you get a stack trace? https://urbackup.atlassian.net/wiki/spaces/US/pages/8323075/Debugging+with+gdb+on+Linux

3

I don’t think the upgrade from 2.1.19 to 2.1.20 has anything to do with the error, it was only for information about which version I was running.

I recently lost the data (not the database), so all the backups had to re-run fully. I ran database repair first, so it would flush from database the backups which were gone, and start over. There is 50+ clients, most of them connected via internet. We have a 500Mbps internet connection, it shows some load, but nothing the server can’t handle (HP G9 32GB RAM, Xeon 8-core).

I started configuring from source, and the configuration showed missing packages, zlib, libcurl and cryptodev. I’ve installed these, and give the pre-compiled version one more go. If it fails again, I’ll do the stack trace.

It has failed the 3 last days after 23.9h service uptime. Right before 24h, the service stops. I’ll either to the stack trace tomorrow, or let you know if the missing packages was the issue.

4

I have the same problem. It will be fine for weeks and then seg fault twice in one day.
If you just need the backup service to restart automatically use immortal from immortal.run

5

It has run for 5 days without any trouble, so this issue can be closed now.

I installed the following packages which were missing:

apt install libcrypto++-dev libcurl4-gnutls-dev zlib1g-dev