UrBackup Server 2.0.8/Client 2.0.7 beta

uroni · March 30, 2016, 6:11pm

Most of the changes are described in the blog:

The Linux command line interfaces have been redesigned since then as well.

This version now includes a Mac OS X client and a portable Linux client (for x86/x86_64/ARMv6/ARM64).
The portable Linux client doesn’t have C+±exception support (yet). In my tests everything was functional but this is something that must be resolved before releasing a non-beta version.

Changes with server 2.0.8/client 2.0.7 beta

Treat file list timeout as timeout and don’t back off
Catch abort and invalid parameters via crashdump on Windows
Fix compression and encrytion bug which caused internet image backups to fail
No error if snapshots are not configured on Linux
Exclude swapfile.sys on Windows
Fixed bug in sparse file backup with “block differences - hashed” transfer mode

Todo

UEFI/GPT testing
Document and fix LDAP/AD login
Reproduce and fix assertion failure described in UrBackup Server/Client 2.0.5 beta (it might have been the incremental sparse file backup bug)

Compatibility with prior versions

2.x server with 1.4.x client full compatibility (please report issues)
2.x client with 1.4.x server works only in local network mode (not via internet mode)
Older client/server combinations may work but were not tested
1.x restore does not work with 2.x servers (improved login method)

Upgrade process

As always: Replace the executables and the database of the server/client will be updated on first running it. As always downgrading the database version after upgrading it is not possible, so you should backup the old database files especially since this is a beta.

Because of the improved file deduplication and statistics calculation the largest server table has to be completely rebuild. This may take a few hours depending on how many file entries you have. It will show the progress on the web interface but is not usable during the upgrade process.

Linux notes:

The wrapper scripts start_urbackup_server and start_urbackup_client have been removed. Please use the executable directly
The executable has been renamed to urbackupsrv (from urbackup_srv), the client to urbackupclientbackend (from urbackup_client)
There is a new command line interface for the client urbackupclientctl
All the plugins are now statically linked into one executable. This simplifies the compilation, debugging and packaging on Linux

Run the UrBackup server on Linux with e.g. urbackupsrv run --loglevel debug

Downloads

Server: https://limitingfactor.urbackup.org/Server/2.0.8%20beta/
Client: https://limitingfactor.urbackup.org/Client/2.0.7%20beta/
Restore: https://limitingfactor.urbackup.org/Restore/2.0.1/
WIP admin manual: admin_guide.pdf (363.3 KB) (not updated)

john3354 · March 30, 2016, 8:08pm

Loaded up the latest and testing with LAN and Internet clients right now. I will let you know how things turn out.

ressel · March 31, 2016, 8:05am

I have updated to 2.0.8 server and on my windows 10 to 2.0.7 client.
Now I get this error:

ERROR: Client calculated hash of "/media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/C/Users/sr/AppData/Local/Google/Chrome/User Data/Default/GPUCache/data_3" differs from server calculated hash. This may be caused by a bug or by random bit flips on the client or server hard disk. Failing backup.
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150214_223402.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150418_213243.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150517_101850.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150604_101853.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150604_102142.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150612_193941.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150624_101114.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150630_224533.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150707_124412.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150716_203702.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150716_211145.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150716_223304.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150928_215458.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20151008_121232.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-0902/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/VID_20150828_203753.mp4
ERROR: Fatal error during backup. Backup not completed
ERROR: FATAL: Backup failed because of disk problems

Dagon · March 31, 2016, 8:19am

Hi @uroni

First of all many thanks for the awesome work you are doing!

I’m mainly interested in the LDAP/AD authentication part testing. I have a setup with OpenLDAP so I assume that some changes are needed for the default LDAP settings, but I’ll come back to those a bit later after investigating.

The first issue that I have with 2.0.8 (and earlier versions) is the compilation on CentOS/RHEL 7:

DVARDIR=’"/usr/local/var"’ -DBINDIR=’"/usr/local/bin"’ -DDATADIR=’"/usr/local/share"’ -g -O2 -MT fsimageplugin/urbackupsrv-cowfile.o -MD -MP -MF fsimageplugin/.deps/urbackupsrv-cowfile.Tpo -c -o fsimageplugin/urbackupsrv-cowfile.o test -f 'fsimageplugin/cowfile.cpp' || echo './'fsimageplugin/cowfile.cpp
In file included from fsimageplugin/fs/ntfs.h:2:0,
from fsimageplugin/cowfile.cpp:30:
fsimageplugin/fs/…/filesystem.h:20:7: warning: ‘Filesystem’ has a field ‘Filesystem::readahead_thread’ whose type uses the anonymous namespace [enabled by default]
class Filesystem : public IFilesystem
^
fsimageplugin/cowfile.cpp: In member function ‘bool CowFile::setUnused(_i64, _i64)’:
fsimageplugin/cowfile.cpp:470:27: error: ‘FALLOC_FL_PUNCH_HOLE’ was not declared in this scope
int rc = fallocate64(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, unused_start, unused_end-unused_start);
^
fsimageplugin/cowfile.cpp:470:48: error: ‘FALLOC_FL_KEEP_SIZE’ was not declared in this scope
int rc = fallocate64(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, unused_start, unused_end-unused_start);

To overcome this I just quickly added the following line to fsimageplugin/cowfile.cpp:

#include “/usr/include/linux/falloc.h”

There’s probably a cleaner way to do this, but I didn’t spend too much time on investigating …

Dagon · March 31, 2016, 8:24am

@uroni

The next issue will occur during “make install” (again on CentOS 7). The syntax for adduser is incompatible, instead of:

adduser urbackup --system --quiet --group …

it should be:

adduser urbackup --system --user-group

CentOS/RHEL version of adduser doesn’t understand flags “–quiet” and “–group”. “–quiet” can be removed, but “–group” should be changed to “–user-group”.

uroni · March 31, 2016, 11:32am

Is this a full backup, or a incremental backup based on a backup done via the last version?

ressel · March 31, 2016, 1:24pm

incremental backup

uroni · March 31, 2016, 2:10pm

Could you run a full file backup and see if the problem still occurs afterwards during incrementals?

ressel · March 31, 2016, 3:55pm

This helped, no errors in log only warnings

ressel · March 31, 2016, 9:23pm

No, now I get this errors again:

ERROR: Client calculated hash of "/media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/C/Users/sr/AppData/Local/Google/Chrome/User Data/Default/Application Cache/Cache/data_2" differs from server calculated hash. This may be caused by a bug or by random bit flips on the client or server hard disk. Failing backup.
ERROR: Client calculated hash of "/media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/C/Users/sr/AppData/Local/Google/Chrome/User Data/Default/Application Cache/Cache/data_3" differs from server calculated hash. This may be caused by a bug or by random bit flips on the client or server hard disk. Failing backup.
ERROR: Client calculated hash of "/media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/C/Users/sr/AppData/Local/Google/Chrome/User Data/Default/GPUCache/data_3" differs from server calculated hash. This may be caused by a bug or by random bit flips on the client or server hard disk. Failing backup.
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150516_103940.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150621_195544_nopm_.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150716_224241.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150721_214255.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150802_150139.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150903_210716.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150909_195345.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20151017_201540.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20151031_103310.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160331-2315/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/Screenshot_20151114-104617.png
ERROR: Fatal error during backup. Backup not completed
ERROR: FATAL: Backup failed because of disk problems

uroni · March 31, 2016, 9:41pm

This is a new server install (not upgrade) of a UrBackup server >= 2.0.5 (tree hashing) and the client uses “block differences - hashed” transfer mode?

ressel · March 31, 2016, 10:34pm

New 2.0.6

Then Upgrade since
Im using default settings. Will check what it is tomorrow

ressel · April 1, 2016, 9:01am

Im using internet mode, so block difference right?
Advanced settings:

Client Settings:

uroni · April 1, 2016, 3:32pm

Could you check if the issue is fixed with this version: https://limitingfactor.urbackup.org/Server/2.0.9%20beta/

ressel · April 1, 2016, 4:01pm

Working on a full backup, and will try incremental later.

ressel · April 1, 2016, 5:37pm

Hello,
First I installed the .dpkg upgrade on my server, then I ran full file backup:

Then I ran incremental backup:
ERROR: Client calculated hash of “/media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/C/Users/sr/AppData/Local/Google/Chrome/User Data/Default/GPUCache/data_3” differs from server calculated hash. This may be caused by a bug or by random bit flips on the client or server hard disk. Failing backup.
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150214_223402.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150418_213243.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150517_101850.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150604_101853.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150604_102142.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150612_193941.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150624_101114.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150630_224533.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150707_124412.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150716_203702.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150716_211145.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150716_223304.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20150928_215458.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/IMG_20151008_121232.jpg
WARNING: Error adding sparse file entry. Could not read hash from /media/backup/urbackup//BornData - SÃ¸ren Yoga 13/160401-1926/.hashes/C/Windows.old/Users/SÃ¸ren/OneDrive/Billeder/Filmrulle/VID_20150828_203753.mp4
ERROR: Fatal error during backup. Backup not completed
ERROR: FATAL: Backup failed because of disk problems

uroni · April 2, 2016, 2:49pm

Thanks for testing! Reproduced and fixed this problem now. Updated in place; should work now: https://limitingfactor.urbackup.org/Server/2.0.9%20beta/

draga · April 3, 2016, 9:25am

I will test it as soon as possible.

ressel · April 3, 2016, 1:59pm

Thank you, this solved the problem.
I will continue testing for other errors.