Server 2.5.4 WIP/Client 2.5.2 WIP

Testing
,
1

New settings handling

The major change with server 2.5.x and client 2.5.x is a complete redesign how settings are handled. The goal is to make it more explicit and consistent.

  • Each setting has a button next to it which switches between inheriting the setting from the group, using the configuration value on the configuration page and using the value from the client, and for some setting merging client and server settings together
  • File backup paths are now a normal setting just like any other. No more separation between paths to backup configured on the client and default directories to backup

This is a critical area of UrBackup and as such needs a lot of testing. Focus should be on if the 2.4.x client still more or less works with the 2.5.x server (and especially doesn’t loose its connection settings for Internet servers).

Linux image backups

2.5.x can backup Linux clients (via dattobd only). The use case is e.g. backing up a DigitalOcean instance. It is explicitly not to create image backups of all possible configurations e.g. LVM, mdraid, btrfs, …
It can “live restore” a root volume, i.e., overwrite the root volume of a currently running machine. So one can image backup e.g. a DigitalOcean droplet. To restore create a new instance (tested with Debian 10/buster) then run the restore script which overwrites the current instance with the backed up instance using a trick to put the currently running system files into (compressed) memory.
Dattobd supports Changed Block Tracking (CBT), so image (and file) backups are done with CBT.

Linux image backup howto

Backup:

  • Install dattobd https://github.com/datto/dattobd/blob/master/INSTALL.md
  • To only backup used space install partclone (e.g. apt install partclone). This is currently supported for ext* and xfs. If partclone is not present it’ll backup unused space.
  • Install the (binary) client and select dattobd for snapshotting

Restore:

  • On the server browse to the image backup to restore
  • Click on “Restore Linux image”
  • Copy & paste the command into a new instance and follow the instructions (the command has a timeout (token), so after a while it should not work anymore)

Testing (image backup + file backup with CBT)

To test if CBT works correctly with file backups please enable Debugging: Verify file backups using client side hashes (for Internet clients with client-side hashing) or Debugging: End-to-end verification of all file backups in the advanced settings.

To test image backups enable md5sums creation on the client via touch /usr/local/var/create_md5sums_imagebackup. The client will then create a file (with date+time in the name) at /usr/local/var/md5sums-* for each image backup listing all files in the image plus their md5sum. Copy that file to server, mount the image backup then verify the md5sums with cd /backup/client/image_mnt; md5sums --quiet -c /path/to/md5sums-file.txt. The only files with checksum errors should be the .datto-*/.overlay-* files in the root directory and swap files (those get automatically excluded from backup).

Thanks for any help with testing and feedback!

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Place the files from the update directory into C:\Program Files\UrBackupServer\urbackup or /var/urbackup to auto-update clients. Disable Download client from update server in the server settings to prevent the server from downloading the current version.

On Linux e.g. with this update script: https://github.com/ptempier/get_urbackupclient/blob/master/updateclient.sh

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.

Changes with server 2.5.4 beta

  • Update SQLite
  • Fix JavaScript issue with settings
  • Fix db recovery (db repair script)

Changes with server 2.5.3 beta

  • Fix db upgrade issue

Changes with server 2.5.2 beta

  • Increase timeouts during backup starts from 10s to 60s
  • Multi-threaded image compression + compress image with ZSTD now
  • Forward image capa to alert script and use it to switch instead of os info
  • Backup and restore grub stage 1.5
  • Make cleanup log message less ambiguous
  • Check if client name is correct before starting backup and offline client if not
  • Upgrade LMDB to version 0.9.24
  • Use MDB_WRITEMAP and disable MDB_NOMETASYNC
  • Don’t encrypt server restore identity when starting restore from web interface
  • Fix some issues with the web interface for the new settings handling
  • Reset r_online when exiting reconnect loop in image backup

Changes with server 2.5.1 beta

  • Changes to support (limited) Linux image backups
  • Fixes to new settings handling
  • Upgrade to Visual Studio 2019
  • Use getrandom instead of reading from /dev/urandom if possible
  • Return error message if authentication key is empty
  • Fix check if image backup is already queued
  • Handle “all_nonusb” setting in image snapshot group
  • Fix storing alert state when serialization has null bytes
  • Fix backup_dirs_optional display + saving
  • Substract sparse extents before checking if file fits on storage
  • Upgrade miniz
  • Fix polish language name
  • Setting to only allow access to settings after the user enters a text
  • Add option to send mail SSL encrypted (without STARTTLS)

Changes with client 2.5.2 beta

  • Remove outdated log message for token not found
  • Fix problem in fix for piped tar file hash issue after resume
  • Check if client name is correct before starting backup and offline client if not
  • Don’t encrypt server restore identity when starting restore from web interface
  • Improve killing processes with files open for write when restoring to root device on Linux

Changes with client 2.5.1 beta

  • Changes to support (limited) Linux image backups
  • Upgrade to Visual Studio 2019
  • Use getrandom instead of reading from /dev/urandom if possible
  • Fix rare race condition that makes indexing hange after reconnect

Changes with server 2.5.0 beta

  • Rework settings handling

Changes with client 2.5.0 beta

  • Rework settings handling

Downloads

3

Hi… I have just tried server 2.5.2 on Debian 10 and there appears to be a database problem. Initally the web interface displayed a message saying the internal database was being upgraded and now the web interface will not load.

The logs show the following.

> 2020-03-28 09:44:20: WARNING: Upgrading...
> 2020-03-28 09:44:21: WARNING: SQLite: no such column: value_client in "SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?" errorcode: 1
> 2020-03-28 09:44:21: ERROR: Error preparing Query [SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?]: no such column: value_client. Retrying in 1s...
> 2020-03-28 09:44:22: WARNING: SQLite: no such column: value_client in "SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?" errorcode: 1
> 2020-03-28 09:44:22: ERROR: Error preparing Query [SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?]: no such column: value_client. Retrying in 1s...
> 2020-03-28 09:44:23: WARNING: SQLite: no such column: value_client in "SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?" errorcode: 1
> 2020-03-28 09:44:23: ERROR: Error preparing Query [SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?]: no such column: value_client. Retrying in 1s...
> 2020-03-28 09:44:24: WARNING: SQLite: no such column: value_client in "SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?" errorcode: 1
> 2020-03-28 09:44:24: ERROR: Error preparing Query [SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?]: no such column: value_client. Retrying in 1s...
> 2020-03-28 09:44:25: WARNING: SQLite: no such column: value_client in "SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?" errorcode: 1
> 2020-03-28 09:44:25: ERROR: Error preparing Query [SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?]: no such column: value_client. Retrying in 1s...
> 2020-03-28 09:44:26: WARNING: SQLite: no such column: value_client in "SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?" errorcode: 1
> 2020-03-28 09:44:26: ERROR: Error preparing Query [SELECT value, value_client, use FROM settings_db.settings WHERE clientid=? AND key=?]: no such column: value_client

If I run urbackupsrv repair-database I get the following.

2020-03-28 09:43:38: SQLite: recovered 10010 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2020-03-28 09:43:38: SQLite: recovered 35943 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2020-03-28 09:43:38: SQLite: recovered 102 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2020-03-28 09:43:38: SQLite: recovered 14 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2020-03-28 09:43:38: SQLite: recovered 10010 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2020-03-28 09:43:38: Recovering contents of database with id 20...
2020-03-28 09:43:38: WARNING: SQLite: API call with invalid database connection pointer errorcode: 21
2020-03-28 09:43:38: WARNING: SQLite: misuse at line 157315 of [c20a353364] errorcode: 21
2020-03-28 09:43:38: WARNING: SQLite: API call with invalid database connection pointer errorcode: 21
2020-03-28 09:43:38: WARNING: SQLite: misuse at line 157315 of [c20a353364] errorcode: 21
Segmentation fault
4

Right, downgraded back to 2.14.2, restored the databases and everything works.
stopped the server and upgraded to 2.5.2
restarted the server and the first reported error returned.

5

Thanks 2.5.3 hopefully fixes the db upgrade issue. I’ll look into the db repair issue later. Thanks for testing!

6

Yep, the server is now working. Thanks

7

Still Testing - There appears to be a problem with the archive function and the save function in general - All my archive settings for all my clients have gone

If I try to make changes to any setting on any tab for a specific client the interface locks up, I can see the progress bar updating but the interface never comes back unless I refresh the page and the setting is not saved.
I can make and save changes to the settings when the drop down is set to Clients.

If I create groups I can edit the settings and click save without the lockup occurring, but nothing is saved.

There is nothing in the logs.

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.