Segfault on client with 2.5.16 and dattobd

On Debian 10 the client throws a segfault when trying to do a full file backup using dattobd snapshot.
The client is 2.5.16, the server 2.5.22

● urbackupclientbackend.service - UrBackup Client backend
   Loaded: loaded (/lib/systemd/system/urbackupclientbackend.service; enabled; v
   Active: failed (Result: signal) since Mon 2021-08-23 17:13:55 CEST; 24s ago
  Process: 5261 ExecStart=/usr/local/sbin/urbackupclientbackend --config /etc/de
 Main PID: 5261 (code=killed, signal=SEGV)

dmesg tells me:
[ 148.725222] file indexing[5266]: segfault at 28 ip 00000000004ab867 sp 00007f22ea7f9730 error 6 in urbackupclientbackend[400000+524000]

I have even switched the client to debug mode:
2021-08-23 17:16:14: Trying to transition /dev/datto0 to snapshot...
2021-08-23 17:16:14: dbdctl transition-to-snapshot '//.datto_3d41c58e-6724-4d47-8981-11c766a08a24_d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b' 0
2021-08-23 17:16:14: Transitioned /dev/datto0 to snapshot.
2021-08-23 17:16:14: Mounting /dev/mapper/wsnap-d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b...
2021-08-23 17:16:14: Shadowcopy path: /mnt/urbackup_snaps/d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b
2021-08-23 17:16:14: Using datto change information from //.datto_3d41c58e-6724-4d47-8981-11c766a08a24_01cc60a567535bbc17bebeed1c7e45dd7a955611380a33d3
2021-08-23 17:16:14: done.
2021-08-23 17:16:14: Zeroing file hash data of volume /...
2021-08-23 17:16:14: Indexing "lib"...
2021-08-23 17:16:14: Hashing file "/mnt/urbackup_snaps/d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b/var/lib/apt/extended_states"

The same problem occurs on multiple clients with different paths / files (always directly after starting indexing).

I also tried using strace when starting the client.

Using datto change information from //.datto_3d41c58e-6724-4d47-8981-11c766a08a24_d4bb103327fe602ae20171a6c50dec262c2575d76cb6e76b
Zeroing file hash data of volume /...
NULL)  = 0
nanosleep({tv_sec=1, tv_nsec=0}, Indexing "lib"...
Hashing file "/mnt/urbackup_snaps/8e88a1b55e429c77eb9aa8e5323cb1142bfbc342ae63e168/var/lib/apt/extended_states"
 <unfinished ...>) = ?
+++ killed by SIGSEGV +++
Segmentation fault

The parallel hashing setting (beta) is disabled. Reboot of server does not help.

Thank you for help.

P.S.: Image backup seems to work correctly, full and incremental. It’s just happening on file backup.

I have now by accident found more information.

All clients previously had a full image backup completed.
When I start the client, umount the datto snapshot manually and then start full backup, it says
2021-08-23 19:44:07: ERROR: driver returned an error performing specified action. check dmesg for more info: Device or resource busy 2021-08-23 19:44:07: ERROR: Using /dev/datto0...

but it does not crash with SEGV and runs successfully (of course with warnings about changed files).

The .datto* and .overlay* files in / seem to remain after crash and even after restart and reboot (don’t know if they are somewhen cleaned up) and fill the disk sooner or later.

Is there any possibility to get further information about the code part it crashes in? Strace does not seem to give useful information and I did not get gdb to run the client successfully :slightly_frowning_face:

Install the * client (for debug information). Then attach gdb to the client by running gdb, then attach PID. Once it crashed bt shows the line it crashes on.

@uroni does that help?

Thread 6 "file indexing" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fbf9164b700 (LWP 6624)]
_ZN10ClientHash12getShaBinaryERKSsR9IHashFuncb (this=0x0, fn=..., hf=..., 
    with_cbt=<optimized out>) at urbackupclient/ClientHash.cpp:156
156	urbackupclient/ClientHash.cpp: No such file or directory.
(gdb) bt
#0  _ZN10ClientHash12getShaBinaryERKSsR9IHashFuncb (this=0x0, fn=..., hf=..., 
    with_cbt=<optimized out>) at urbackupclient/ClientHash.cpp:156
#1  0x00000000004efc21 in getShaBinary (with_cbt=<optimized out>, hf=..., 
    fn=..., this=0x23c0180) at urbackupclient/client.cpp:5522
#2  getShaBinary (this=this@entry=0x23c0180, fn=...)
    at urbackupclient/client.cpp:5501
#3  0x00000000004f17b5 in addMissingHashes (calc_hashes=true, 
    include_dirs=..., exclude_dirs=..., namedpath=..., filepath=..., 
    orig_path=..., fsfiles=0x7fbf91649d38, dbfiles=0x7fbf91649810, 
    this=0x23c0180) at urbackupclient/client.cpp:3127
#4  _ZN11IndexThread13getFilesProxyERKSsSsS1_bS1_bRKSt6vectorISsSaISsEERKS2_I13SIndexIncludeSaIS7_EERx (this=this@entry=0x23c0180, orig_path=..., 
    path=<incomplete type>, named_path=..., use_db=use_db@entry=false, 
    fn_filter=..., use_db_hashes=true, exclude_dirs=..., include_dirs=..., 
    target_generation=@0x7fbf91649ca8: 5) at urbackupclient/client.cpp:3311
#5  0x00000000005da5df in initialCheck (this=this@entry=0x23c0180, 
    params_stack=..., stack_idx=stack_idx@entry=18446744073709551615, 
    volume=..., vssvolume=..., orig_dir=<incomplete type>, 
    dir=<incomplete type>, named_path=<incomplete type>, outfile=..., 
    first=true, flags=44, use_db=true, symlinked=false, depth=1, 
    dir_recurse=true, include_exclude_dirs=true, exclude_dirs=..., 
    include_dirs=..., orig_path=...) at urbackupclient/client.cpp:2513
#6  0x00000000005df8b2 in indexDirs (this=this@entry=0x23c0180, 
--Type <RET> for more, q to quit, c to continue without paging--
    _backup@entry=false, simultaneous_other=<optimized out>) at urbackupclient/client.cpp:1886
#7  0x00000000005e18ef in _ZN11IndexThreadclEv (this=0x23c0180) at urbackupclient/client.cpp:868
#8  0x00000000005cc0ee in thread_helper_f (t=0x23c0180) at Server.cpp:1487
#9  0x00007fbf93825fa3 in start_thread () from /lib/x86_64-linux-gnu/
#10 0x00007fbf937564cf in clone () from /lib/x86_64-linux-gnu/

Once I disable the filesystem snapshot entries in the snapshot.cfg, the backups run without problems. The volume_snapshot entries in snapshot.cfg are still there and image backup is working without crashes.