Segfault on client with 2.5.16 and dattobd

client
1

On Debian 10 the client throws a segfault when trying to do a full file backup using dattobd snapshot.
The client is 2.5.16, the server 2.5.22

● urbackupclientbackend.service - UrBackup Client backend
   Loaded: loaded (/lib/systemd/system/urbackupclientbackend.service; enabled; v
   Active: failed (Result: signal) since Mon 2021-08-23 17:13:55 CEST; 24s ago
  Process: 5261 ExecStart=/usr/local/sbin/urbackupclientbackend --config /etc/de
 Main PID: 5261 (code=killed, signal=SEGV)

dmesg tells me:
[ 148.725222] file indexing[5266]: segfault at 28 ip 00000000004ab867 sp 00007f22ea7f9730 error 6 in urbackupclientbackend[400000+524000]

I have even switched the client to debug mode:
2021-08-23 17:16:14: Trying to transition /dev/datto0 to snapshot...
2021-08-23 17:16:14: dbdctl transition-to-snapshot '//.datto_3d41c58e-6724-4d47-8981-11c766a08a24_d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b' 0
2021-08-23 17:16:14: Transitioned /dev/datto0 to snapshot.
2021-08-23 17:16:14: Mounting /dev/mapper/wsnap-d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b...
2021-08-23 17:16:14: Shadowcopy path: /mnt/urbackup_snaps/d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b
2021-08-23 17:16:14: Using datto change information from //.datto_3d41c58e-6724-4d47-8981-11c766a08a24_01cc60a567535bbc17bebeed1c7e45dd7a955611380a33d3
2021-08-23 17:16:14: done.
2021-08-23 17:16:14: Zeroing file hash data of volume /...
2021-08-23 17:16:14: Indexing "lib"...
2021-08-23 17:16:14: Hashing file "/mnt/urbackup_snaps/d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b/var/lib/apt/extended_states"

The same problem occurs on multiple clients with different paths / files (always directly after starting indexing).

I also tried using strace when starting the client.

Using datto change information from //.datto_3d41c58e-6724-4d47-8981-11c766a08a24_d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b
done.
Zeroing file hash data of volume /...
NULL)  = 0
nanosleep({tv_sec=1, tv_nsec=0}, Indexing "lib"...
Hashing file "/mnt/urbackup_snaps/8e88a1b55e429c77eb9aa8e5323cb1142bfbc342ae63e168/var/lib/apt/extended_states"
 <unfinished ...>) = ?
+++ killed by SIGSEGV +++
Segmentation fault

The parallel hashing setting (beta) is disabled. Reboot of server does not help.

Thank you for help.

P.S.: Image backup seems to work correctly, full and incremental. It’s just happening on file backup.

2

I have now by accident found more information.

All clients previously had a full image backup completed.
When I start the client, umount the datto snapshot manually and then start full backup, it says
2021-08-23 19:44:07: ERROR: driver returned an error performing specified action. check dmesg for more info: Device or resource busy 2021-08-23 19:44:07: ERROR: Using /dev/datto0...

but it does not crash with SEGV and runs successfully (of course with warnings about changed files).

The .datto* and .overlay* files in / seem to remain after crash and even after restart and reboot (don’t know if they are somewhen cleaned up) and fill the disk sooner or later.

3

Is there any possibility to get further information about the code part it crashes in? Strace does not seem to give useful information and I did not get gdb to run the client successfully :slightly_frowning_face:

4

Install the *-dbg.sh client (for debug information). Then attach gdb to the client by running gdb, then attach PID. Once it crashed bt shows the line it crashes on.

5

@uroni does that help?

Thread 6 "file indexing" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fbf9164b700 (LWP 6624)]
_ZN10ClientHash12getShaBinaryERKSsR9IHashFuncb (this=0x0, fn=..., hf=..., 
    with_cbt=<optimized out>) at urbackupclient/ClientHash.cpp:156
156	urbackupclient/ClientHash.cpp: No such file or directory.
(gdb) bt
#0  _ZN10ClientHash12getShaBinaryERKSsR9IHashFuncb (this=0x0, fn=..., hf=..., 
    with_cbt=<optimized out>) at urbackupclient/ClientHash.cpp:156
#1  0x00000000004efc21 in getShaBinary (with_cbt=<optimized out>, hf=..., 
    fn=..., this=0x23c0180) at urbackupclient/client.cpp:5522
#2  getShaBinary (this=this@entry=0x23c0180, fn=...)
    at urbackupclient/client.cpp:5501
#3  0x00000000004f17b5 in addMissingHashes (calc_hashes=true, 
    include_dirs=..., exclude_dirs=..., namedpath=..., filepath=..., 
    orig_path=..., fsfiles=0x7fbf91649d38, dbfiles=0x7fbf91649810, 
    this=0x23c0180) at urbackupclient/client.cpp:3127
#4  _ZN11IndexThread13getFilesProxyERKSsSsS1_bS1_bRKSt6vectorISsSaISsEERKS2_I13SIndexIncludeSaIS7_EERx (this=this@entry=0x23c0180, orig_path=..., 
    path=<incomplete type>, named_path=..., use_db=use_db@entry=false, 
    fn_filter=..., use_db_hashes=true, exclude_dirs=..., include_dirs=..., 
    target_generation=@0x7fbf91649ca8: 5) at urbackupclient/client.cpp:3311
#5  0x00000000005da5df in initialCheck (this=this@entry=0x23c0180, 
    params_stack=..., stack_idx=stack_idx@entry=18446744073709551615, 
    volume=..., vssvolume=..., orig_dir=<incomplete type>, 
    dir=<incomplete type>, named_path=<incomplete type>, outfile=..., 
    first=true, flags=44, use_db=true, symlinked=false, depth=1, 
    dir_recurse=true, include_exclude_dirs=true, exclude_dirs=..., 
    include_dirs=..., orig_path=...) at urbackupclient/client.cpp:2513
#6  0x00000000005df8b2 in indexDirs (this=this@entry=0x23c0180, 
--Type <RET> for more, q to quit, c to continue without paging--
    _backup@entry=false, simultaneous_other=<optimized out>) at urbackupclient/client.cpp:1886
#7  0x00000000005e18ef in _ZN11IndexThreadclEv (this=0x23c0180) at urbackupclient/client.cpp:868
#8  0x00000000005cc0ee in thread_helper_f (t=0x23c0180) at Server.cpp:1487
#9  0x00007fbf93825fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fbf937564cf in clone () from /lib/x86_64-linux-gnu/libc.so.6

Once I disable the filesystem snapshot entries in the snapshot.cfg, the backups run without problems. The volume_snapshot entries in snapshot.cfg are still there and image backup is working without crashes.