UrBackup Server 2.1.14 beta/Client 2.1.11 beta

Testing
21

Yes Debian package. How should i set it up ? Installing the dbg package ?

22

Run gdb in screen attach to pid and then run generate-core-file once it crashed.

23

Ok it works with gdb and then attach <PID> but after few minutes, the program “hangs” until i quit gdb and detach the debugging session …

(gdb) attach 12431
Attaching to process 12431
Reading symbols from /usr/bin/urbackupsrv...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libcurl.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcurl.so.4
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libfuse.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libfuse.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 15522]
[New LWP 15521]
[New LWP 15044]
[New LWP 15037]
[New LWP 15032]
[New LWP 15026]
[New LWP 15025]
[New LWP 14912]
[New LWP 14678]
[New LWP 14677]
[New LWP 14676]
[New LWP 14674]
[New LWP 14266]
[New LWP 14234]
[New LWP 14150]
[New LWP 14118]
[New LWP 14034]
[New LWP 14033]
[New LWP 13815]
[New LWP 13691]
[New LWP 13677]
[New LWP 13664]
[New LWP 13488]
[New LWP 13487]
[New LWP 13486]
[New LWP 13485]
[New LWP 13484]
[New LWP 13483]
[New LWP 13482]
[New LWP 13444]
[New LWP 13432]
[New LWP 13375]
[New LWP 13374]
[New LWP 13362]
[New LWP 13273]
[New LWP 13264]
[New LWP 13263]
[New LWP 13262]
[New LWP 13261]
[New LWP 13260]
[New LWP 13227]
[New LWP 13226]
[New LWP 12690]
[New LWP 12687]
[New LWP 12674]
[New LWP 12673]
[New LWP 12669]
[New LWP 12647]
[New LWP 12646]
[New LWP 12643]
[New LWP 12642]
[New LWP 12641]
[New LWP 12640]
[New LWP 12639]
[New LWP 12638]
[New LWP 12631]
[New LWP 12626]
[New LWP 12613]
[New LWP 12609]
[New LWP 12608]
[New LWP 12607]
[New LWP 12606]
[New LWP 12605]
[New LWP 12604]
[New LWP 12603]
[New LWP 12602]
[New LWP 12601]
[New LWP 12600]
[New LWP 12572]
[New LWP 12523]
[New LWP 12522]
[New LWP 12521]
[New LWP 12520]
[New LWP 12519]
[New LWP 12518]
[New LWP 12517]
[New LWP 12516]
[New LWP 12515]
[New LWP 12514]
[New LWP 12513]
[New LWP 12512]
[New LWP 12488]
[New LWP 12482]
[New LWP 12481]
[New LWP 12480]
[New LWP 12479]
[New LWP 12478]
[New LWP 12477]
[New LWP 12476]
[New LWP 12452]
[New LWP 12451]
[New LWP 12450]
[New LWP 12449]
[New LWP 12448]
[New LWP 12447]
[New LWP 12446]
[New LWP 12445]
[New LWP 12444]
[New LWP 12443]
[New LWP 12435]
[New LWP 12434]
[New LWP 12433]
[New LWP 12432]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libidn.so.11
Reading symbols from /usr/lib/x86_64-linux-gnu/librtmp.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/librtmp.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libssh2.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssh2.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcom_err.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28
Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhogweed.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libnettle.so.4
Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgmp.so.10
Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcrypt.so.20
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libkeyutils.so.1
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libsasl2.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libtasn1.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgpg-error.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libffi.so.6
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2
0x00007f0ca52f6aed in poll () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)  quit
A debugging session is active.

        Inferior 1 [process 12431] will be detached.

Quit anyway? (y or n) y
Detaching from program: /usr/bin/urbackupsrv, process 12431
root@backup-2:~#
24

Well, you have to type in continue for it to continue…

25

Is there a way to automate this, because i’ll not stay in front of my screen all over the night :confused: …

26

What do you mean automate? Do you think you have to type in continue all the time?

27

Yes …

28

If you want to do that, you should type in next or step not continue :wink:

29

Reproduce the problem, but cannot generate core dump :

(gdb) continue
Continuing.
[New Thread 0x7f0c3a1a9700 (LWP 17345)]
[New Thread 0x7f0c41c4c700 (LWP 17346)]
[New Thread 0x7f0c40c4a700 (LWP 17354)]
[New Thread 0x7f0c3a9aa700 (LWP 17361)]
[New Thread 0x7f0c4c7e8700 (LWP 17362)]
[New Thread 0x7f0c4bfe7700 (LWP 17363)]
[Thread 0x7f0c3a9aa700 (LWP 17361) exited]
[Thread 0x7f0c40c4a700 (LWP 17354) exited]
[New Thread 0x7f0c40c4a700 (LWP 17880)]
[New Thread 0x7f0c3a9aa700 (LWP 17881)]
[New Thread 0x7f0c4b7e6700 (LWP 17882)]
[Thread 0x7f0c4bfe7700 (LWP 17363) exited]
[Thread 0x7f0c4c7e8700 (LWP 17362) exited]
[Thread 0x7f0c3a1a9700 (LWP 17345) exited]
[Thread 0x7f0c40c4a700 (LWP 17880) exited]
[New Thread 0x7f0c40c4a700 (LWP 17961)]
[New Thread 0x7f0c3a1a9700 (LWP 17962)]
[New Thread 0x7f0c4c7e8700 (LWP 17963)]
[New Thread 0x7f0c4bfe7700 (LWP 17964)]
[New Thread 0x7f0c4afe5700 (LWP 17965)]
[New Thread 0x7f0c4a7e4700 (LWP 17966)]
[New Thread 0x7f0c49fe3700 (LWP 17979)]
[New Thread 0x7f0c497e2700 (LWP 18100)]
[New Thread 0x7f0c48fe1700 (LWP 18107)]
[New Thread 0x7f0c487e0700 (LWP 18112)]
[New Thread 0x7f0c47fdf700 (LWP 18113)]
[Thread 0x7f0c49fe3700 (LWP 17979) exited]
[Thread 0x7f0c4a7e4700 (LWP 17966) exited]
[New Thread 0x7f0c4a7e4700 (LWP 20078)]
[Thread 0x7f0c4a7e4700 (LWP 20078) exited]
[New Thread 0x7f0c4a7e4700 (LWP 20079)]
[Thread 0x7f0c3a1a9700 (LWP 17962) exited]
[Thread 0x7f0c487e0700 (LWP 18112) exited]
[Thread 0x7f0c47fdf700 (LWP 18113) exited]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f0c48fe1700 (LWP 18107)]
0x00007f0ca524c067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) generate-core-file
warning: target file /proc/12431/cmdline contained unexpected null characters
Couldn't get registers: Aucun processus de ce type.
(gdb)
30

Try writing it somewhere else (see help generate-core-file) with generate-core-file /tmp/core.file…

31

If it does not work with gdb, there are other ways to get a core dump.

32

It doesn’t work with gdb …

33

How should i get a core dump other than with gdb ? gcore <PID> ? Installing the -dbg package ?

Regards,

34

The stack overflow question I linked has the other method as best answer. It can also be configured in the systemd unit or by installing and configuring something like apport.

35

Running process in terminal gives more informations :

2017-01-26 20:31:00: Trim beyond drivesize (drivesize: 128846396928 trim to 128846397440). Trimming less...
2017-01-26 20:31:00: Starting trimming image file (if possible)
2017-01-26 20:31:39: Authed+capa for client 'SV407' (compressed-v2, token auth) - 1 spare connections
2017-01-26 20:32:40: Authed+capa for client 'SV407' (compressed-v2, token auth) - 1 spare connections
2017-01-26 20:33:41: Authed+capa for client 'SV407' (compressed-v2, token auth) - 1 spare connections
2017-01-26 20:33:50: Trimmed 2.58545 GB
*** Error in `/usr/bin/urbackupsrv': free(): invalid next size (normal): 0x00007f40c4160e70 ***
36

Unfortunately that does not give me more information to track down this bug.

37

I’m able to reproduce the problem at every backup but i’m not able to produce a core dump.
I’ve tried with gdb and also with setting ulimit -c unlimited and starting urbackupsrv in CLI …

Perhaps you have other suggestions ?

Regards,

38

Have you set the core pattern?

39

Alternatively you can also compile it from sources then run it in gdb and enter bt once it crashed.

40

Yes :

root@backup-2:~# cat /proc/sys/kernel/core_pattern 
/cores/core.%e.%p.%t
root@backup-2:~# ll / | grep cores
drwxrwxrwx   2 root root  4096 janv. 25 19:18 cores
root@backup-2:~#