Server 2.3.2 beta/Client 2.3.1 beta/Restore 2.2.1 beta


#1

Major changes with server 2.3.x beta

  • Use reflinks on file systems that do not have snapshot/subvolume support (only XFS for now)
  • Run backup storage checks asynchronously from status screen
  • Allow client renames in some cases
  • Preparations for special Hyper-V client with disk images instead of volume images
  • Allow client to restore its own image backups (image restore with normal client)

Changes with server 2.3.2 beta

  • Load all lua modules per default
  • Do not strip debug information (line numbers) when compiling lua script
  • Comment to disable lua code compilation

Changes with server 2.3.1 beta

  • Functionality to log file changes in incremental image backups
  • Seek to correct position in source file before sending when resuming image restore
  • Return error correctly when server runs out of space during file backup in a corner case
  • Only use CURLOPT_EXPECT_100_TIMEOUT_MS if curl version is high enough
  • Fix Crypto++ 6.0 byte compile issue
  • Prevent cleanup from deleting incremental image chains that cause number of incremental images to fall below min_incr_image (in testing)

Changes with server 2.3.0 beta

  • Improve space accounting if reflinking is replaced by copying
  • Alert/status fixes
  • Set json content type when returning json in web interface
  • Stop parallel hash download thread in all cases
  • Parallel file hashing fixes
  • Add Pulseway alert script (mainly as an example)
  • Improve out of space error handling
  • Log login attempts where username does not exist as failed login attempts
  • Fix settings threading issue
  • Log failed client authentications into auth log (Linux)

Changes with client 2.3.1 beta

  • Fix permission translation for Linux and macOS

Changes with client 2.3.0 beta

  • Retry llistxattr with a multiple of the required memory if it fails
  • Cooperate CBT with multiple running clients
  • Don’t show restore components option if components never have been backed up
  • Fix change indicator of open files such that the symbolic link detection works for them
  • Read USNs of symbolic links and directories on Windows instead of last modified time to detect directory metadata changes
  • Fix file permission translation on Linux/MacOS
  • Fix Crypto++ 6.0 byte compile issue
  • Handle reparse tag IO_REPARSE_TAG_APPEXECLINK on Windows
  • Backup VeraCrypt boot loader
  • Improve log message if indexing is interrupted

Changes with Restore CD 2.2.1 beta

  • Restore whole disks (created with Hyper-V client)
  • Continuously track image download progress for resume
  • Restore VeraCrypt boot loader

Changes with Restore CD 2.2.0 beta

  • Bugfix: In some cases the restore did not continue at the most recent position when it had to reconnect to the server
  • Build with default Debian stretch live-build and not the accumulation of patches, fixes and manual changes I had beforehand

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Place the files from the update directory into C:\Program Files\UrBackupServer\urbackup or /var/urbackup to auto-update clients. Disable Download client from update server in the server settings to prevent the server from downloading the current version.

On Linux e.g. with this update script: https://github.com/ptempier/get_urbackupclient/blob/master/updateclient.sh

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.

Downloads


Execute shell command from alert script
Server 2.3.1 beta/Client 2.3.0 beta
#2

#3

Getting failed image backups both with server beta 2.3.1 and 2.3.2, both against matching beta clients, in a way similar to kiga’s report. All failures are on Internet clients, but only 3 of 5 so far. The other two clients worked for the most recent image incremental under server 2.3.1.
Settings are the same for both succeeding and failing Internet clients, which are branch office Windows Server 2008r2 with very similar contents. Both incremental and (Internet style) synthetic full backups fail.
Local clients (Win7pro) haven’t reported any new problems so far; both full image and incrementals succeed, and local full file and incrementals succeed.

Typical client error log on server, from the first test of 2.3.2:

Info    10/13/18 22:48   Starting unscheduled full image backup of volume "C:"...
Info    10/13/18 22:48   Basing image backup on last incremental or full image backup
Errors  10/13/18 22:49   Error opening VHD file "E:\UrBackup\Schertz\181013-2248_Image_C\Image_C_181013-2248.vhdz"
Info    10/13/18 22:49   Transferred 6.06271 MB - Average speed: 1.53269 MBit/s
Info    10/13/18 22:49   Time taken for backing up client Schertz: 35s
Errors  10/13/18 22:49   Backup failed

And the matching text from server urbackup.log:

2018-10-13 22:49:24: ERROR: Error opening VHD file
2018-10-13 22:49:24: ERROR: Error opening Parentvhdfile "E:\UrBackup\Schertz/180929-2014_Image_C/Image_C_180915-1603.vhdz"
2018-10-13 22:49:24: ERROR: Error opening Parentvhdfile "E:\UrBackup\Schertz/180929-2014_Image_C/Image_C_180929-2014.vhdz"
2018-10-13 22:49:24: ERROR: Error opening parent VHD
2018-10-13 22:49:24: ERROR: Error opening VHD file "E:\UrBackup\Schertz\181013-2248_Image_C\Image_C_181013-2248.vhdz"
2018-10-13 22:49:24: ERROR: Backup failed
2018-10-13 22:49:25: WARNING: Exponential backoff: Waiting at least 40m before next image backup

Addendum: I see no differences in file security properties for the saved Image_C files between succeeding and failing clients to account for the failure to open.


#4

Does E:\UrBackup\Schertz/180915-1603_Image_C/Image_C_180915-1603.vhdz exist and if not, do you have an idea when it might have gotten deleted?


#5

Would have sworn that’s the one I checked permissions on last night, but it isn’t there now, nor is it listed on the Backups tab for Schertz. No mention of that file in app.log or urbackup.log until the recent error opening message, and both start before 180915. [I searched the logs for “180915” which is in the filename.] The debug log posted earlier was created just for that one test. I also do a nightly Windows Backup, but the UrBackup storage path (E:\UrBackup) is excluded so no help there.

From time to time I’ve seen the latest incremental file backups deleted automatically, but don’t recall it happening with images. Trying to fix that is why I’m trying the 2.3.x beta versions here instead of the 2.2.11 release used previously. 2.3.0 was installed July 4, 2.3.1 on September 27, and 2.3.2 on October 13.

Trying to open any of the later image backups crashes urbackup_srv.exe, as pasted from Event Viewer below. The earlier (and only remaining) full image of C from 180908 opens immediately. The other two images, incrementals from 180929 and 181006, both fail to open. I have the most recent 631MB crash dump file saved, if that’s useful.

I can delete the Schertz client and files if a clean slate is needed. I’d rather not wipe the whole server install and 3.2TB of backups but that can be done if necessary.

Excerpt from C:\Program Files\UrBackupServer\urbackup.log:

2018-10-14 10:19:46: ERROR: Error opening VHD file
2018-10-14 10:19:46: ERROR: Error opening Parentvhdfile "E:\UrBackup\Schertz\180929-2014_Image_C/Image_C_180915-1603.vhdz"
2018-10-14 10:19:48: ERROR: Error opening VHD file
2018-10-14 10:19:48: ERROR: Error opening Parentvhdfile "E:\UrBackup\Schertz\180929-2014_Image_C/Image_C_180915-1603.vhdz"
2018-10-14 10:19:48: ERROR: Error opening VHD file
2018-10-14 10:19:48: ERROR: Error opening Parentvhdfile "E:\UrBackup\Schertz\180929-2014_Image_C/Image_C_180915-1603.vhdz"
2018-10-14 10:19:48: ERROR: Fatal exception code 3221225477 at address 0x000007FEE282D329
2018-10-14 10:19:49: ERROR: Fatal exception (APPLICATION CRASHED). Crash dump written to "C:\Windows\TEMP\UrBackup\v2.0.0-20181014-101948-4220-5356.dmp"
2018-10-14 10:19:49: ERROR: Program abort (SIGABRT)

From Event Viewer:

Faulting application name: urbackup_srv.exe, version: 0.0.0.0, time stamp: 0x5ba010f8
Faulting module name: KERNELBASE.dll, version: 6.1.7601.24260, time stamp: 0x5b9470f4
Exception code: 0x00000000
Fault offset: 0x000000000000bded
Faulting process id: 0x107c
Faulting application start time: 0x01d463d1323a30d9
Faulting application path: C:\Program Files\UrBackupServer\urbackup_srv.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 971231e6-cfc4-11e8-8c2f-6c0b843ef24f

#6

Thanks, could reproduce the problem. Server 2.3.3 beta/Restore 2.2.2 beta probably fixes it (not the crash though), though you’ll have to run a new full image backup.


#7

Thank you for the update. With 2.3.3 synthetic full image still fails with same file not found issue. Going to painfully slow (using 10Mbit uplink) real full works under 2.3.2 and 2.3.3. Will check tonight if incrementals work.