Out of a sudden, backup failed

Talkabout · May 2, 2021, 4:14pm

Hi guys,

I have several Linux clients where I do a regular (daily) file backup of the complete file system. This worked fine for all of them for months, but suddenly one of those clients failed to backup with the following message:

I have checked the file on the client and it exists. I have no space issue on the server side. Other clients are backing up correctly. I did a full backup and this also works fine. I even deleted all backups and started from scratch, but as soon as I start an incremental file backup it fails. Any chance I can fix that?

The log file on the server provides following information:

2021-05-02 15:42:53: Backing up "root" without snapshot.                                                         
2021-05-02 15:42:53: Indexing of "root" done. 3543 filesystem lookups 0 db lookups and 0 db updates              
2021-05-02 15:42:53: Backing up "boot" without snapshot.                                                         
2021-05-02 15:42:53: Indexing of "boot" done. 3 filesystem lookups 0 db lookups and 0 db updates                 
2021-05-02 15:42:53: keybar.home.intern: Loading file list...                                                    
2021-05-02 15:42:54: keybar.home.intern: Calculating file tree differences...                                    
2021-05-02 15:42:55: keybar.home.intern: Calculating tree difference size...                                     
2021-05-02 15:42:55: keybar.home.intern: Linking unchanged and loading new files...                              
2021-05-02 15:42:55: Referencing snapshot on "keybar.home.intern" for path "root" failed: FAILED                 
2021-05-02 15:43:17: WARNING: Error getting complete file "NABSDzvPNYBnB1dzHCsK|root/usr/local/var/urbackup/data/
filelist_new_0.ub" from keybar.home.intern. Errorcode: CANNOT_OPEN_FILE (3)                                      
2021-05-02 15:43:38: Socket has error: 111                                                                       
2021-05-02 15:43:39: ERROR: Error getting file metadata. Errorcode: CANNOT_OPEN_FILE (3)                         
2021-05-02 15:43:51: ERROR: Error getting complete file "NABSDzvPNYBnB1dzHCsK|root/usr/share/man/cs/man8/aptitude
.8.gz" from keybar.home.intern. Errorcode: BASE_DIR_LOST (7)                                                     
2021-05-02 15:43:51: Hint: Current root backup path does not exist anymore. Usually this is because the current f
ile system snapshot it was backing up was deleted because it ran out of snapshot storage space. See https://www.u
rbackup.org/faq.html#base_dir_lost for details and for how to fix this issue                                     
2021-05-02 15:43:51: ERROR: Client keybar.home.intern went offline.                                              
2021-05-02 15:44:02: Waiting for file transfers...                                                               
2021-05-02 15:44:08: Waiting for file hashing and copying threads...                                             
2021-05-02 15:44:39: Writing new file list...                                                                    
2021-05-02 15:44:41: Transferred 8.29976 MB - Average speed: 1.20838 MBit/s                                      
2021-05-02 15:44:41: Time taken for backing up client keybar.home.intern: 2m 3s                                  
2021-05-02 15:44:41: ERROR: Backup failed

Thanks!

Bye

Talkabout · May 2, 2021, 4:30pm

This “id” is the same for every incremental backup I start on the affected client, regardless of how many other backups are available for it.

Bye

Talkabout · May 2, 2021, 6:28pm

I have realized one thing: when trying to execute “urbackupclientbackend --help” I get the following error at the end of the list of possible commands:

I am not seeing this on any of my other clients. In the log I can see a line like

2021-05-02 20:16:06: ERROR: Error getting file type of /usr/local/var/urbackup/data/filelist_new_0.ub

After that the client is not reachable any more, which will explain the socket error. “journalctl” gives following output:

How to fix this?

Thanks!

Bye

Talkabout · May 3, 2021, 3:30pm

Hi all,

I started a debug run on the client, but the last lines do not give any hint on what goes wrong:

...
ERROR: Error getting file type of /usr/local/var/urbackup/data/filelist_new_0.ub
...
FileSrv: Received data...
FileSrv: Received a Packet.
FileSrv: Sending meta data of NABSDzvPNYBnB1dzHCsK|root/usr/share/doc/python2.7/python-policy.sgml.gz metadata_id=23679
FileSrv: Mapped name: /usr/share/doc/python2.7/python-policy.sgml.gz
FileSrv: Received a Packet.
FileSrv: Sending meta data of NABSDzvPNYBnB1dzHCsK|root/usr/share/doc/python2.7/python-policy.txt.gz metadata_id=23680
FileSrv: Mapped name: /usr/share/doc/python2.7/python-policy.txt.gz
FileSrv: Received a Packet.
FileSrv: Sending meta data of NABSDzvPNYBnB1dzHCsK|root/usr/share/doc/python3.4/changelog.gz metadata_id=23727
FileSrv: Mapped name: /usr/share/doc/python3.4/changelog.gz
FileSrv: Received a Packet.
Received flush.
ClientService cmd: #IevUiP9V71ljdftLPuuVf#2PING RUNNING pc_done=39&status_id=26&speed_bpms=0.035293&total_bytes=7671905&done_bytes=3024662&paused_fb=1#token=NAB
SDzvPNYBnB1dzHCsK
xxx@keybar:~ $

How to debug further? Is the ERROR relevant? It is not the last line (there are 50-100 lines after) but still is something that worries me.

Thanks!

Bye

uroni · May 3, 2021, 3:54pm

What’s your client version + system architecture?

Talkabout · May 3, 2021, 4:08pm

Client: raspberry pi zero w2, raspbian jessie (8)

UrBackup Client Backend v2.4.11.0
Copyright (C) 2011-2019 Martin Raiber
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Raspbian is installed on a SD Card. Raspberry is connected via WIFI (2.4 GHz).

Any other info you need?

Thanks!

uroni · May 3, 2021, 4:13pm

Seems the precompiled version isn’t compatible with your CPU. Compile it yourself?

Talkabout · May 3, 2021, 4:16pm

The backups were working for months without issues. I have installed the client based on what is provided on the home page:

UrBackup - Download UrBackup for Windows, GNU/Linux or FreeBSD

Talkabout · May 3, 2021, 4:18pm

Interesting: when running install script I see following output:

Verifying archive integrity... All good.
Uncompressing UrBackup Client Installer for Linux  100%
Installation of UrBackup Client 2.4.11 to /usr/local ... Proceed ? [Y/n]
Y
Uncompressing install data...
Detected Debian (derivative) system
Detected systemd
Detected architecture arm-linux-androideabi
Illegal instruction
(Android NDK build not working. Falling back to ELLCC build...)
Installing systemd unit...
Restarting UrBackup Client service...
Successfully started client service. Installation complete.

Can this be a possible reason? Would an update to Raspbian Buster help?

Talkabout · May 4, 2021, 7:10pm

I updated to Raspbian Buster (10) but the issue persists. It is really strange why a full backup runs without problems but an incremental backup fails with a crash of the urbackup client. Any hints how I can dig deeper into that issue?

Thanks!

Talkabout · June 3, 2021, 4:15pm

I have compiled urbackup client now manually on that system and the error seems to be gone.