File backup issue (Linux client) on Server 2.5.22

Description: I’m trying to perform full file backup from the system (Centos 7) and it seems that indexing process is failing.

Steps to reproduce:

  1. Setup urbackup linux client and dattobd
  2. Configure for file back. I’m backing up /.
  3. Run full file backup on the server.

Expected result:
Backup is performed successfully

Actual result:
Segfault for client during indexing phase.

Logs:

Feb 10 12:00:07 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:17 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Script "/usr/local/etc/urbackup/prefilebackup" returned with success
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Redirecting to /bin/systemctl stop crond.service
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Final path: /
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Creating shadowcopy of "rootfs" in indexDirs()
Feb 10 12:00:19 URBACKUP_CLIENT kernel: datto: fallocate is not supported for '/.datto_3d41c58e-6724-4d47-8981-11c766a08a24_943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9', falling back on writing zeros
Feb 10 12:00:27 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:37 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:47 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:51 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: PONG
Feb 10 12:00:57 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): mounting ext3 file system using the ext4 subsystem
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): orphan cleanup on readonly fs
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): 1 orphan inode deleted
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): recovery complete
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Snapshotting device /dev/sdb via dattobd...
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Trying to transition /dev/datto0 to snapshot...
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: dbdctl transition-to-snapshot '//.datto_3d41c58e-6724-4d47-8981-11c766a08a24_943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9' 0
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Transitioned /dev/datto0 to snapshot.
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Mounting /dev/mapper/wsnap-943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9...
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Shadowcopy path: /mnt/urbackup_snaps/943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Using datto change information from //.datto_3d41c58e-6724-4d47-8981-11c766a08a24_65f7d63d6c4400cdc94cb02dac9920e20a6fb33afc4b30d3
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: done.
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Zeroing file hash data of volume /...
Feb 10 12:01:07 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:17 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:27 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:37 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:47 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:51 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: PONG
Feb 10 12:01:57 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:07 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:09 URBACKUP_CLIENT urbackupclientbackend[9204]: Indexing "rootfs"...
Feb 10 12:02:10 URBACKUP_CLIENT urbackupclientbackend[9204]: Not following symlink "/etc/mtab" because symlink target at "/proc/9204/mounts" is excluded
Feb 10 12:02:17 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:27 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:37 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:47 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:48 URBACKUP_CLIENT kernel: phash[29806]: segfault at 8 ip 00000000004b34de sp 00007fc10213f740 error 4 in urbackupclientbackend[400000+524000]
Feb 10 12:02:48 URBACKUP_CLIENT kernel: Code: ff e8 2e 26 ff ff e9 0c 03 00 00 31 f6 3d 10 02 00 00 48 89 df 0f 85 5d 01 00 00 e8 56 6a 02 00 49 8b 86 38 01 00 00 4c 89 ff <48> 8b 70 08 48 85 f6 48 0f 45 f0 e8 92 45 0b 00 49 8b be 38 01 00
Feb 10 12:02:48 URBACKUP_CLIENT systemd[1]: urbackupclientbackend.service: main process exited, code=killed, status=11/SEGV
Feb 10 12:02:48 URBACKUP_CLIENT systemd[1]: Unit urbackupclientbackend.service entered failed state.
Feb 10 12:02:48 URBACKUP_CLIENT systemd[1]: urbackupclientbackend.service failed.

Switching off parallel hashing brings this segfault:

file indexing[30842]: segfault at 28 ip 00000000004ab867 sp 00007f524a302750 error 6 in urbackupclientbackend[400000+524000]

The rest is the same.

Will be glad to help to pin it down. I have no issues backing up in “file mode” from other linux servers (also Centos 7), so it seems that this server or volume (ext3) is somehow special.

Might be connected with the CBT. Could you install the version with debug symbols (UrBackup Client Linux 2.5.17-dbg.sh) and run it in gdb ?

E.g. gdb --args urbackupclientbackend -v debug, then run, then bt once it crashes (or generate a core dump and send me that).

This issue seems to be resolved with new client/server.