Server 2.5.9 beta (updated x3)

New settings handling

The major change with server 2.5.x and client 2.5.x is a complete redesign how settings are handled. The goal is to make it more explicit and consistent.

  • Each setting has a button next to it which switches between inheriting the setting from the group, using the configuration value on the configuration page and using the value from the client, and for some setting merging client and server settings together
  • File backup paths are now a normal setting just like any other. No more separation between paths to backup configured on the client and default directories to backup

This is a critical area of UrBackup and as such needs a lot of testing. Focus should be on if the 2.4.x client still more or less works with the 2.5.x server (and especially doesn’t loose its connection settings for Internet servers).

Linux image backups

2.5.x can backup Linux clients (via dattobd only). The use case is e.g. backing up a DigitalOcean instance. It is explicitly not to create image backups of all possible configurations e.g. LVM, mdraid, btrfs, …
It can “live restore” a root volume, i.e., overwrite the root volume of a currently running machine. So one can image backup e.g. a DigitalOcean droplet. To restore create a new instance (tested with Debian 10/buster) then run the restore script which overwrites the current instance with the backed up instance using a trick to put the currently running system files into (compressed) memory.
Dattobd supports Changed Block Tracking (CBT), so image (and file) backups are done with CBT.

Linux image backup howto

Backup:

  • Install dattobd https://github.com/datto/dattobd/blob/master/INSTALL.md
  • To only backup used space install partclone (e.g. apt install partclone). This is currently supported for ext* and xfs. If partclone is not present it’ll backup unused space.
  • Install the (binary) client and select dattobd for snapshotting

Restore:

  • On the server browse to the image backup to restore
  • Click on “Restore Linux image”
  • Copy & paste the command into a new instance and follow the instructions (the command has a timeout (token), so after a while it should not work anymore)

Testing (image backup + file backup with CBT)

To test if CBT works correctly with file backups please enable Debugging: Verify file backups using client side hashes (for Internet clients with client-side hashing) or Debugging: End-to-end verification of all file backups in the advanced settings.

To test image backups enable md5sums creation on the client via touch /usr/local/var/create_md5sums_imagebackup. The client will then create a file (with date+time in the name) at /usr/local/var/md5sums-* for each image backup listing all files in the image plus their md5sum. Copy that file to server, mount the image backup then verify the md5sums with cd /backup/client/image_mnt; md5sums --quiet -c /path/to/md5sums-file.txt. The only files with checksum errors should be the .datto-*/.overlay-* files in the root directory and swap files (those get automatically excluded from backup).


Thanks for any help with testing and feedback!

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Place the files from the update directory into C:\Program Files\UrBackupServer\urbackup or /var/urbackup to auto-update clients. Disable Download client from update server in the server settings to prevent the server from downloading the current version.

On Linux e.g. with this update script: https://github.com/ptempier/get_urbackupclient/blob/master/updateclient.sh

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.


On testing beta/WIP versions

Please see Having problems with UrBackup? Please read before posting for what kind of information to add when reporting issues with the beta/WIP version. Reports such as “X does not work” are useless. In general, at the very least, think about what kind of information one would need to reproduce the problem in a different environment. Also a reminder that this is a community project which depends on your contribution and help. Testing is a way to contribute, but without detailed reports on the issues you are finding, you might as well wait for the final release. Best case you debug the problem yourself and then post the fix.

I’ll ignore any issue reports that don’t follow minimum reporting standards.


Changes with server 2.5.9 beta

  • Set unused in bitmap on Windows with cow files (ReFS)
  • Recompile cURL with SSL support for Windows
  • Improve cURL error handling

Changes with server 2.5.8 beta

  • Fix settings upgrade

Changes with server 2.5.7 beta

  • Improve/fix settings upgrade
  • Fix merged settings display
  • Fix merged setting validation causing JavaScript error

Changes with server 2.5.6 beta

  • Update client CAPA after client restart
  • Handle new zfs command line version return
  • Use reflink if hard links aren’t available (Fixes file backup to ReFS)
  • Don’t try to set restore settings when downloading Windows client from web interface
  • Fix deleting raw images if backup storage does not support snapshots
  • Also send _def key for backward compatibility with 2.4.x clients
  • Limit size of in-memory log to 2MiB per backup
  • Copy items from lost+found to files table in db repair
  • Create new subvolume with name extented with .startup-del (when using btrfs) to prevent race in cleanup
  • Improve LMDB file index performance by spilling pages earlier and by disabling MDB_WRITEMAP again
  • Merge LMDB memory leak fix

Changes with server 2.5.5 beta

  • Fix blockalign reading/writing to stdin/stdout
  • Make last modification time configurable from script list
  • Fix backup of symlinks to directories with special characters

Changes with server 2.5.4 beta

  • Update SQLite
  • Fix JavaScript issue with settings
  • Fix db recovery (db repair script)

Changes with server 2.5.3 beta

  • Fix db upgrade issue

Changes with server 2.5.2 beta

  • Increase timeouts during backup starts from 10s to 60s
  • Multi-threaded image compression + compress image with ZSTD now
  • Forward image capa to alert script and use it to switch instead of os info
  • Backup and restore grub stage 1.5
  • Make cleanup log message less ambiguous
  • Check if client name is correct before starting backup and offline client if not
  • Upgrade LMDB to version 0.9.24
  • Use MDB_WRITEMAP and disable MDB_NOMETASYNC
  • Don’t encrypt server restore identity when starting restore from web interface
  • Fix some issues with the web interface for the new settings handling
  • Reset r_online when exiting reconnect loop in image backup

Changes with server 2.5.1 beta

  • Changes to support (limited) Linux image backups
  • Fixes to new settings handling
  • Upgrade to Visual Studio 2019
  • Use getrandom instead of reading from /dev/urandom if possible
  • Return error message if authentication key is empty
  • Fix check if image backup is already queued
  • Handle “all_nonusb” setting in image snapshot group
  • Fix storing alert state when serialization has null bytes
  • Fix backup_dirs_optional display + saving
  • Substract sparse extents before checking if file fits on storage
  • Upgrade miniz
  • Fix polish language name
  • Setting to only allow access to settings after the user enters a text
  • Add option to send mail SSL encrypted (without STARTTLS)

Changes with client 2.5.3 beta

  • Enable script backup for sub-clients
  • Fix mariadb backup by using mbstream format
  • Ignore backup GPT errors and use primary twice instead on error
  • Fix “ALL” file backup dirs handling

Changes with client 2.5.2 beta

  • Remove outdated log message for token not found
  • Fix problem in fix for piped tar file hash issue after resume
  • Check if client name is correct before starting backup and offline client if not
  • Don’t encrypt server restore identity when starting restore from web interface
  • Improve killing processes with files open for write when restoring to root device on Linux

Changes with client 2.5.1 beta

  • Changes to support (limited) Linux image backups
  • Upgrade to Visual Studio 2019
  • Use getrandom instead of reading from /dev/urandom if possible
  • Fix rare race condition that makes indexing hange after reconnect

Changes with server 2.5.0 beta

  • Rework settings handling

Changes with client 2.5.0 beta

  • Rework settings handling

Downloads

2 Likes

Just installed the beta update on a Windows 2012 server.
Seems like I’m running into the same problem than stated here : Server 2.5.4 WIP/Client 2.5.2 WIP

Most of my clients lost their archive settings, and they also try to backup another folder than the folder specified in the settings of their parent groups.

Got the same issue:

  • updated from UrBackup Server 2.4.12 to 2.5.6 beta on Windows Server 2019
  • tried to change settings on client level
  • after hitting “save” the interface locks up, i.e. the spinner runs forever and the progress bar never finishes. After reloading the page, the settings were not saved.

Uncommon: only one out of two clients is affected. After updating to 2.5.6 beta, client1 showed the redesigned config button on all settings right away and works as expected. But client2 has the old settings layout without the redesigned config button. As soon as hitting save for the first time on a settings page of client2, the redesigned buttons appear but the interface is locked (see screenshots below).

Client version: 2.4.10-cbt
The issue seems to be independent of the clients: I’ve restored a backup of the UrBackup Server to a testing environment without any network connection and reproduced the bug.
Unfortunately, nothing is shown in the logs (debug level).

solution: remove the entire client, run cleanup, begin from scratch with new client.

other steps tried so far without success:

  • rebooted
  • run cleanup.bat, cleanup_database.bat, repair_database.bat, remove_unknown.bat

client1 with new config buttons right away after update to 2.5.6 beta:

client2 does not show the new config buttons after update to 2.5.6 beta:

after hitting save for the first time, the new config buttons appear but the interface is locked and settings are not saved:

If you are getting javascript errors on the web interface, please open the web developer console (CTRL+SHIFT+I) and copy+paste errors from the console here. Thanks!

2020-06-10 js error on save

Digged a bit deeper:
restored backup of Server 2.4.12 and created 5 new clients before trying to update to 2.5.6 beta.

Results after updating to 2.5.6 beta:

working as expected:
“clientUntouched”: client created but no separate settings were configured - not even enabled separate settings once.

not able to save settings (js error):
“clientSeparateSettingsNoChanges”: client created and separate settings activated but no settings were changed

“clientSeparateSettingsWithChanges”: client created, separate settings activated and changed the setting “Maximale Anzahl an inkrementellen Dateisicherungen”.

“clientSeparateSettingsNoChangesReverted”: client created, separate settings activated but deactived right away without any settings change

“clientSeparateSettingsWithChangesReverted”: client created, separate settings activated, changed setting “Maximale Anzahl an inkrementellen Dateisicherungen”, disabled separate settings afterwards

result:
The bug occurs as soon as separate settings were enabled once.

There is another null exception when loading a config page of client which got separate settings enabled once. Only the first new redesigned config button is shown (see screenshots below).

TypeError thrown when loading settings page of client (first part) and error thrown on clicking save button (red part):
2020-06-10 js error before save

Perhaps 2.5.7 fixes some of the settings issues.

Unfortunately, Urbackup Server won’t start after the Update from 2.4.12 to 2.5.7:

urbackup.log: (level debug)
2020-06-10 23:12:08: WARNING: Upgrading…
2020-06-10 23:12:08: WARNING: Upgrading database to version 60
2020-06-10 23:12:08: WARNING: SQLite: near “DATE”: syntax error in “DATE settings_db.settings SET use=” errorcode: 1
2020-06-10 23:12:08: ERROR: Error preparing Query [DATE settings_db.settings SET use=]: near “DATE”: syntax error
2020-06-10 23:12:08: ERROR: Upgrading database failed. Shutting down server.


compared to log when updating from 2.4.12 to 2.5.6:
2020-06-10 23:19:40: WARNING: Upgrading…
2020-06-10 23:19:40: WARNING: Upgrading database to version 60
2020-06-10 23:19:40: WARNING: Upgrading database to version 61
2020-06-10 23:19:40: WARNING: Upgrading database to version 62
2020-06-10 23:19:40: WARNING: Done.

In 2.5.7 beta might something be broken at line 2093 in dllmain.cpp:

Seems to be ok on github, but probably not up to date yet.

Sorry, should be fixed now.

Yes, everything is working as expected with Server 2.5.8 beta so far. Many thanks for your great work!

1 Like

There might be another bug in current 2.5.x beta:
After the update, sending notification mail fails:

log:
2020-06-12 13:20:55: WARNING: Error sending mail to “log@{private}”. Login denied(ec=67), . Retrying in 30m

os: Windows Server 2019

Mail config:
Mailservername: email-smtp.eu-west-1.amazonaws.com
Mailserverport: 587
Mailserver-Benutzername: {private}
Mailserver-Passwort: {private}
Absender-E-Mail-Adresse: log@{private}
E-Mails nur mit SSL/TLS-Verschlüsselung versenden: yes
SSL/TLS-Zertifikat überprüfen: no
Use SSL encrypted SMTP (SMTPS) instead of SMTP with STARTTLS: no

Trying to send a test mail on config page shows:
Senden der Test-E-Mail fehlgeschlagen. Fehler: Login denied(ec=67),

Rolled back to 2.4.12: works.

Steps tried so far: reset all mail config without success.

The cURL it uses to send mail was upgraded to 7.68. Perhaps you can try sending mail with the curl from https://curl.haxx.se/windows/dl-7.68.0/ to confirm this is an issue with cURL ( https://stackoverflow.com/a/16069786/506718 ) ?

Probably this issue?

Using the following curl line works as expected:

curl --url "smtp://email-smtp.eu-west-1.amazonaws.com:587" --ssl-reqd --mail-from "log@{fromMail}" --mail-rcpt "testmail@{toMail}" --upload-file mail.txt --user "{username}:{password}" --verbose

content of mail.txt:


From: log <log@{fromMail}>
To: test <test@{toMail}>
Subject: Sending Using Curl

This is a curl test mail.


Successfully tested with curl versions:
7.68.0-win32
7.68.0-win64
7.70.0-win32
7.70.0-win64

The error “Login denied(ec=67)” is shown if the username/password is invalid or the --ssl-reqd switch is missing (ssl-reqd corresponds to curl_easy_setopt option CURLOPT_USE_SSL set to CURLUSESSL_ALL).

I’ve looked through the code for changes to “ssl_only” switch. There was only a minor change on 2020/01/13 when implementing smtps:

before: ms.ssl_only=(settings->getValue("mail_ssl_only", "false")=="true")?true:false;
after: ms.ssl_only = settings->getValue("mail_ssl_only", "false")=="true";

I’d say both statements evaluate to the same and therefore shouldn’t matter.

Any further ideas, e.g. is it possible to enable curl verbose logging in UrBackup?

Ok, so AWS SES (luckily) requires STARTTLS, but the new Windows cURL was accidentally compiled without SSL support. The new version fixes this and improves the error handling so this doesn’t occur again.

This could be classified as a security issue because it results in unencrypted connections, but the cURL with missing SSL was only used in the 2.5.x betas.

Tested and working: 2.5.9 brings back STARTTLS.
Thanks for the fix!

I’ve been doing quite a bit testing and maybe there is some minor inconsistency with the new settings handling:

1.) The “Archive”-settings default to custom settings (no inheritance) while other settings default to “use settings from group”:

expected:


Maybe there is a reason for that and its not a bug.


2.) On config page “Alarms” only the first field is inherited. All others don’t show the new settings button.


Not sure, whether on purpose or not yet implemented.

It’s the switch for all settings (since alert settings are customizable it is internally a single setting). Maybe I’ll add a separator or something to make that more clear. And maybe have the single settings have switches in a future version but that is a lot of work…