Description: I’m trying to perform full file backup from the system (Centos 7) and it seems that indexing process is failing.
Steps to reproduce:
- Setup urbackup linux client and dattobd
- Configure for file back. I’m backing up /.
- Run full file backup on the server.
Expected result:
Backup is performed successfully
Actual result:
Segfault for client during indexing phase.
Logs:
Feb 10 12:00:07 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:17 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Script "/usr/local/etc/urbackup/prefilebackup" returned with success
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Redirecting to /bin/systemctl stop crond.service
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Final path: /
Feb 10 12:00:19 URBACKUP_CLIENT urbackupclientbackend[9204]: Creating shadowcopy of "rootfs" in indexDirs()
Feb 10 12:00:19 URBACKUP_CLIENT kernel: datto: fallocate is not supported for '/.datto_3d41c58e-6724-4d47-8981-11c766a08a24_943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9', falling back on writing zeros
Feb 10 12:00:27 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:37 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:47 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:51 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: PONG
Feb 10 12:00:57 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): mounting ext3 file system using the ext4 subsystem
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): orphan cleanup on readonly fs
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): 1 orphan inode deleted
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): recovery complete
Feb 10 12:00:58 URBACKUP_CLIENT kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Snapshotting device /dev/sdb via dattobd...
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Trying to transition /dev/datto0 to snapshot...
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: dbdctl transition-to-snapshot '//.datto_3d41c58e-6724-4d47-8981-11c766a08a24_943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9' 0
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Transitioned /dev/datto0 to snapshot.
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Mounting /dev/mapper/wsnap-943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9...
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Shadowcopy path: /mnt/urbackup_snaps/943b5df4dda03e1fadeb3c63718bdf04b94f0e9589577ca9
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Using datto change information from //.datto_3d41c58e-6724-4d47-8981-11c766a08a24_65f7d63d6c4400cdc94cb02dac9920e20a6fb33afc4b30d3
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: done.
Feb 10 12:00:58 URBACKUP_CLIENT urbackupclientbackend[9204]: Zeroing file hash data of volume /...
Feb 10 12:01:07 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:17 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:27 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:37 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:47 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:01:51 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: PONG
Feb 10 12:01:57 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:07 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:09 URBACKUP_CLIENT urbackupclientbackend[9204]: Indexing "rootfs"...
Feb 10 12:02:10 URBACKUP_CLIENT urbackupclientbackend[9204]: Not following symlink "/etc/mtab" because symlink target at "/proc/9204/mounts" is excluded
Feb 10 12:02:17 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:27 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:37 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:47 URBACKUP_CLIENT urbackupclientbackend[9204]: ClientService cmd: #If4Zbx61s7YCr97wyRmSR#2PING RUNNING pc_done=&status_id=52&speed_bpms=0&total_bytes=-1&done_bytes=0&paused_fb=1#token=tMvcsnR8c4ReFTm6QnWI
Feb 10 12:02:48 URBACKUP_CLIENT kernel: phash[29806]: segfault at 8 ip 00000000004b34de sp 00007fc10213f740 error 4 in urbackupclientbackend[400000+524000]
Feb 10 12:02:48 URBACKUP_CLIENT kernel: Code: ff e8 2e 26 ff ff e9 0c 03 00 00 31 f6 3d 10 02 00 00 48 89 df 0f 85 5d 01 00 00 e8 56 6a 02 00 49 8b 86 38 01 00 00 4c 89 ff <48> 8b 70 08 48 85 f6 48 0f 45 f0 e8 92 45 0b 00 49 8b be 38 01 00
Feb 10 12:02:48 URBACKUP_CLIENT systemd[1]: urbackupclientbackend.service: main process exited, code=killed, status=11/SEGV
Feb 10 12:02:48 URBACKUP_CLIENT systemd[1]: Unit urbackupclientbackend.service entered failed state.
Feb 10 12:02:48 URBACKUP_CLIENT systemd[1]: urbackupclientbackend.service failed.
Switching off parallel hashing brings this segfault:
file indexing[30842]: segfault at 28 ip 00000000004ab867 sp 00007f524a302750 error 6 in urbackupclientbackend[400000+524000]
The rest is the same.
Will be glad to help to pin it down. I have no issues backing up in “file mode” from other linux servers (also Centos 7), so it seems that this server or volume (ext3) is somehow special.