Server 2.5.23/Client 2.5.17

New settings handling

The major change with server 2.5.x and client 2.5.x is a complete redesign how settings are handled. The goal is to make it more explicit and consistent.

  • Each setting has a button next to it which switches between inheriting the setting from the group, using the configuration value on the configuration page and using the value from the client, and for some setting merging client and server settings together
  • File backup paths are now a normal setting just like any other. No more separation between paths to backup configured on the client and default directories to backup

This is a critical area of UrBackup and as such needs a lot of testing. Also focus should be on if the 2.4.x client still more or less works with the 2.5.x server (and especially doesn’t loose its connection settings for Internet servers).

Connect via web sockets

With 2.5.x clients can connect via web sockets to the server. Setup should be easy. Instead of “Internet server name/IP” of e.g. “example.com” use “ws://example.com:55414/socket”. The real advantage comes into play if a real web server with SSL is setup in front. Either proxy the whole page, or serve the files and only proxy “/x” via FastCGI (see manual), additionally proxy “/socket” via web socket to “ws://127.0.0.1:55414/socket”. Then setup the clients to connect to “Internet server name” “wss://example.com/socket”. Basic authentication, a different name and a different port should also work, e.g. “wss://username:password@example.com/socket”.

On Debian install apache2, then:

a2enmod proxy
a2enmod proxy_wstunnel

then to your apache (SSL) site configuration add e.g.

Alias /urbackup /usr/share/urbackup/www
ProxyPass "/urbackup/x" "fcgi://localhost:55413"
ProxyPass "/urbackup/socket"  "ws://localhost:55414/socket"

Comeback of macOS client

A test version of the macOS client is here. Please add your feedback!

Linux image backups

2.5.x can backup Linux clients (via dattobd or Linux device mapper snapshots). The use case is e.g. backing up a DigitalOcean instance. It is explicitly not to create image backups of all possible configurations e.g. LVM, mdraid, btrfs, …
It can “live restore” a root volume, i.e., overwrite the root volume of a currently running machine. So one can image backup e.g. a DigitalOcean droplet. To restore create a new instance (tested with Debian 10/buster) then run the restore script which overwrites the current instance with the backed up instance using a trick to put the currently running system files into (compressed) memory.
Dattobd and the Linux device mapper supports Changed Block Tracking (CBT), so image (and file) backups are done with CBT.

The Linux device mapper snapshots scripts are currently only tested on Debian 10 and with the root volume. It needs to hook into the boot sequence (initramfs) to backup the root volume. For other volumes it probably needs to hook into udev. Help getting it running on other distributions/with non-root volumes would be welcome!

Linux image backup howto

Backup:

  • To only backup used space install partclone (e.g. apt install partclone). This is currently supported for ext* and xfs. If partclone is not present it’ll backup unused space.
  • Install the (binary) client and select either dattobd or the Linux device mapper for snapshotting
  • If you are using dattobd: Install dattobd https://github.com/datto/dattobd/blob/master/INSTALL.md

Restore:

  • On the server browse to the image backup to restore
  • Click on “Restore Linux image”
  • Copy & paste the command into a new instance and follow the instructions (the command has a timeout (token), so after a while it should not work anymore)

Testing (image backup + file backup with CBT)

To test if CBT works correctly with file backups please enable Debugging: Verify file backups using client side hashes (for Internet clients with client-side hashing) or Debugging: End-to-end verification of all file backups in the advanced settings.

To test image backups enable md5sums creation on the client via touch /usr/local/var/create_md5sums_imagebackup. The client will then create a file (with date+time in the name) at /usr/local/var/md5sums-* for each image backup listing all files in the image plus their md5sum. Copy that file to server, mount the image backup then verify the md5sums with cd /backup/client/image_mnt; md5sums --quiet -c /path/to/md5sums-file.txt. The only files with checksum errors should be the .datto-*/.overlay-* files in the root directory and swap files (those get automatically excluded from backup).


Thanks for any help with testing and feedback!

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.


On testing

Please see Having problems with UrBackup? Please read before posting for what kind of information to add when reporting issues. Reports such as “X does not work” are useless. In general, at the very least, think about what kind of information one would need to reproduce the problem in a different environment. Best case you debug the problem yourself and then post the fix.


Changes with client 2.5.17

  • Fix unintialized error value on chunked file load
  • Ignore case when looking at websocket headers
  • Add uninstall of initramfs script to Linux client uninstall
  • Add switches for some settings to urbackupclientctl set-settings
  • Correctly set client settings if set via command line
  • Add MariaDB per detabase backup (by Alexander Zaitsev)
  • Add panic with proper error message when root device is snapshot on Linux boot
  • Use vcpkg for Windows dependencies

Changes with server 2.5.23

  • Use vcpkg for Windows dependencies
  • Ignore case when looking at websocket headers
  • Fix getting archival prefix + index
  • Fix encoding issues on adding client
  • Show passive settings/correct name in settings tab
  • Fix vhdx entry key/value check
  • Fix vhdx backing file destruction
  • Fix vhdx handling of default number of compression threads
  • Fix connecting to list of servers
  • Use new settings switches when showing client config
  • Add new config options for only binding to localhost to defaults file
  • Add winbtrfs support
  • Fix vhdx restore

Downloads

1 Like

Does this release fix this issue: Server 2.5.22 and VHDX Volumes - #7 by Dark_Angel ?

Yes, this should fix vhdxz + vhdx issues.

Does this release also fix issue: https://forums.urbackup.org/t/server-2-5-22-client-2-5-16-can-not-create-virtual-subclients/10828/2

Yes, this should fix it.

Can’t get clients to autoupdate to 2.5.17

Installed server 2.5.23 with no issues

copied all files in the “install” folder for client 2.5.17 to /var/urbackup as docs instruct and restarted server
clients do not update to 2.5.17 (stay at current version 2.5.16)
when I check /var/urbackup the version.txt file has been changed from 280 (which is what I downloaded and copied over) back to 279???
If I go into the gui and download the (windows) client, it downloads version 2.5.16 instead of 2.5.17???
in existing client program files\urbackup client exe is 2.5.16 and there is no urbackupupdate.exe file?

Also, when I run urbackupudate.exe manually, windows 10 defender blocks running it…not sure if that is the issue or not

You should consider exempting the UrBackup folders from all AntiVirus/AntiMalware

Figured it out. Had to disable “Download Client from Update Server” in server settings. Update was downloading the prior client version. This is a bug that needs to be fixed.

Confirmed fix. Works with VHDX and VHDZ now. Tested on Server (Centos 7) and Client (Windows 10).

Seems like new recovery image (2.4.0) is also working now - previously I was getting no image backups when client was selected during restoration step. Only text console version worked fine. Now both versions do the job.

Also previously VHDX restorations failed with not clear error. Now everything works.

Amazing job fixing ALL this. Thank you!

1 Like

Seems like I was happy too early and responded after tests in test environment.

After rolling out this to production, few days was everything good, but then, I’ve encountered the following issue: after few days VHDX files and directories are simply deleted from the backup server.

Logs for the first time after error:

Apr 14 12:27:03 BACKUP_SERVER urbackupsrv[3210]: Time taken for backing up client BACKUP_CLIENT6: 3m 10s
Apr 14 12:27:03 BACKUP_SERVER urbackupsrv[3210]: Backup completed with issues
Apr 14 12:27:03 BACKUP_SERVER urbackupsrv[3210]: Updating statistics...
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Updating image stats...
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: msg=WAKEUP
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: server_prepare_hash Thread finished (exit)
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: server_hash Thread finished - normal
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_SERVER/220411-2243_Image_C/Image_C_220411-2243.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT1/220411-2301_Image_SYSVOL/Image_SYSVOL_220411-2301.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT1/220411-2301_Image_ESP/Image_ESP_220411-2301.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT1/220411-2302_Image_C/Image_C_220411-2302.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT2/220412-0103_Image__dev_sdb/Image__dev_sdb_220412-0103.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT3/220412-0313_Image_C/Image_C_220412-0313.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT4/220412-1222_Image_SYSVOL/Image_SYSVOL_220412-1222.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT4/220412-1224_Image_ESP/Image_ESP_220412-1224.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT4/220412-1224_Image_C/Image_C_220412-1224.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT5/220412-1801_Image_SYSVOL/Image_SYSVOL_220412-1801.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT5/220412-1801_Image_C/Image_C_220412-1801.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT6/220412-2031_Image_SYSVOL/Image_SYSVOL_220412-2031.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT6/220412-2031_Image_ESP/Image_ESP_220412-2031.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT6/220412-2031_Image_C/Image_C_220412-2031.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_SERVER/220412-2247_Image_C/Image_C_220412-2247.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT2/220413-0107_Image__dev_sdb/Image__dev_sdb_220413-0107.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_CLIENT3/220413-0316_Image_C/Image_C_220413-0316.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Repairing image failed
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: ERROR: Error opening file '/mnt/safecopy/urbackup/BACKUP_SERVER/220413-2247_Image_C/Image_C_220413-2247.vhdx'
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Updating file statistics...
Apr 14 12:27:04 BACKUP_SERVER urbackupsrv[3210]: Done updating statistics.

As you see - multiple clients, multiple dates are affected.

There are no:

  1. Errors on filesystem or server prior to this. Filesystem is ZFS.
  2. No server restarts (OS and Urbackup), no explicit clean up calls.
  3. No errors reported on the FS, Kernel, Hardware or any subsystem level.
  4. Other programs working with the storage, except Urbackup
  5. No other people accessing server

Following missing files are present in Urbackup DB, i.e. visible via web interface → Backups

Issue wasn’t present when VHD files were used.

I have no idea what is leading to this - would be happy to help to pin it. My best guess is that this is somehow related to image repair routine or image clean up routine (my quota set to 98%, but in fact only 64% of space is occupied), since errors start to appear when or due to such routines started.

On statistics chart, those events are visible like so:

As you see, there were 2 drops, i.e. issue happened twice. For the first time I thought that this is some mistake from my side and established additional monitoring over the situation. However it confirmed to be repeating.

Also I’m not able to delete missing images from Web interface, it errors with:

Apr 16 23:43:40 BACKUP_SERVER urbackupsrv[3029]: Deleting image backup ( id=311, path=/mnt/safecopy/urbackup/BACKUP_CLIENT/220412-2031_Image_C/Image_C_220412-2031.vhdx ) ...
Apr 16 23:43:40 BACKUP_SERVER urbackupsrv[3029]: WARNING: Deleting /mnt/safecopy/urbackup/BACKUP_CLIENT/220412-2031_Image_C/Image_C_220412-2031.vhdx failed. No such file or directory (code: 2)
Apr 16 23:43:40 BACKUP_SERVER urbackupsrv[3029]: WARNING: Deleting /mnt/safecopy/urbackup/BACKUP_CLIENT/220412-2031_Image_C/Image_C_220412-2031.vhdx.hash failed. No such file or directory (code: 2)
Apr 16 23:43:40 BACKUP_SERVER urbackupsrv[3029]: WARNING: Deleting /mnt/safecopy/urbackup/BACKUP_CLIENT/220412-2031_Image_C/Image_C_220412-2031.vhdx.mbr failed. No such file or directory (code: 2)
Apr 16 23:43:40 BACKUP_SERVER urbackupsrv[3029]: WARNING: Deleting /mnt/safecopy/urbackup/BACKUP_CLIENT/220412-2031_Image_C/Image_C_220412-2031.vhdx.cbitmap failed. No such file or directory (code: 2)
Apr 16 23:43:40 BACKUP_SERVER urbackupsrv[3029]: WARNING: Deleting /mnt/safecopy/urbackup/BACKUP_CLIENT/220412-2031_Image_C/Image_C_220412-2031.vhdx.sync failed. No such file or directory (code: 2)
Apr 16 23:43:40 BACKUP_SERVER urbackupsrv[3029]: WARNING: Deleting /mnt/safecopy/urbackup/BACKUP_CLIENT/220412-2031_Image_C/Image_C_220412-2031.vhdx.bitmap failed. No such file or directory (code: 2)

And leaves record in place. Recreating files manually (with touch), allows deletion to be completed. Probably records should be deleted in case of manual request if there are no physical files on disk. Just a suggestion.

Until then I will have to revert back to VHD on prod, unfortunately.

Looked into it and one issue that is definitely there is that “remove unknown” erroneously removes vhdx backups. That doesn’t run on its own, tough. Maybe you have a cron job running that?

That is right - I have cron job which runs that task weekly and it seems to be matching the graph drop times.