Server 2.5.12 beta/Client 2.5.6 beta

New settings handling

The major change with server 2.5.x and client 2.5.x is a complete redesign how settings are handled. The goal is to make it more explicit and consistent.

  • Each setting has a button next to it which switches between inheriting the setting from the group, using the configuration value on the configuration page and using the value from the client, and for some setting merging client and server settings together
  • File backup paths are now a normal setting just like any other. No more separation between paths to backup configured on the client and default directories to backup

This is a critical area of UrBackup and as such needs a lot of testing. Focus should be on if the 2.4.x client still more or less works with the 2.5.x server (and especially doesn’t loose its connection settings for Internet servers).

Connect via web sockets

With 2.5.x clients can connect via web sockets to the server. Setup should be easy. Instead of “Internet server name/IP” of e.g. “example.com” use “ws://example.com:55414/socket”. The real advantage comes into play if a real web server with SSL is setup in front. Either proxy the whole page, or serve the files and only proxy “/x” via FastCGI (see manual), additionally proxy “/socket” via web socket to “ws://127.0.0.1:55414/socket”. Then setup the clients to connect to “Internet server name” “wss://example.com/socket”. Basic authentication, a different name and a different port should also work, e.g. “wss://username:password@example.com/socket”.

Linux image backups

2.5.x can backup Linux clients (via dattobd only). The use case is e.g. backing up a DigitalOcean instance. It is explicitly not to create image backups of all possible configurations e.g. LVM, mdraid, btrfs, …
It can “live restore” a root volume, i.e., overwrite the root volume of a currently running machine. So one can image backup e.g. a DigitalOcean droplet. To restore create a new instance (tested with Debian 10/buster) then run the restore script which overwrites the current instance with the backed up instance using a trick to put the currently running system files into (compressed) memory.
Dattobd supports Changed Block Tracking (CBT), so image (and file) backups are done with CBT.

Linux image backup howto

Backup:

  • Install dattobd https://github.com/datto/dattobd/blob/master/INSTALL.md
  • To only backup used space install partclone (e.g. apt install partclone). This is currently supported for ext* and xfs. If partclone is not present it’ll backup unused space.
  • Install the (binary) client and select dattobd for snapshotting

Restore:

  • On the server browse to the image backup to restore
  • Click on “Restore Linux image”
  • Copy & paste the command into a new instance and follow the instructions (the command has a timeout (token), so after a while it should not work anymore)

Testing (image backup + file backup with CBT)

To test if CBT works correctly with file backups please enable Debugging: Verify file backups using client side hashes (for Internet clients with client-side hashing) or Debugging: End-to-end verification of all file backups in the advanced settings.

To test image backups enable md5sums creation on the client via touch /usr/local/var/create_md5sums_imagebackup. The client will then create a file (with date+time in the name) at /usr/local/var/md5sums-* for each image backup listing all files in the image plus their md5sum. Copy that file to server, mount the image backup then verify the md5sums with cd /backup/client/image_mnt; md5sums --quiet -c /path/to/md5sums-file.txt. The only files with checksum errors should be the .datto-*/.overlay-* files in the root directory and swap files (those get automatically excluded from backup).


Thanks for any help with testing and feedback!

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Place the files from the update directory into C:\Program Files\UrBackupServer\urbackup or /var/urbackup to auto-update clients. Disable Download client from update server in the server settings to prevent the server from downloading the current version.

On Linux e.g. with this update script: https://github.com/ptempier/get_urbackupclient/blob/master/updateclient.sh

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.


On testing beta/WIP versions

Please see Having problems with UrBackup? Please read before posting for what kind of information to add when reporting issues with the beta/WIP version. Reports such as “X does not work” are useless. In general, at the very least, think about what kind of information one would need to reproduce the problem in a different environment. Also a reminder that this is a community project which depends on your contribution and help. Testing is a way to contribute, but without detailed reports on the issues you are finding, you might as well wait for the final release. Best case you debug the problem yourself and then post the fix.

I’ll ignore any issue reports that don’t follow minimum reporting standards.


Changes with server 2.5.12 beta

  • Improve compatibility with older clients w.r.t. settings
  • Set current setting as client setting during database upgrade
  • Disable deletion of group archive entries
  • Only validate current settings when saving settings
  • Various new settings handling fixes
  • Add last modified time to each setting to resolve conflicts of which kind of setting should be used (from client or server)
  • For compatibility reasons add a setting for Internet port again and keep non-default setting on upgrade

Changes with server 2.5.11 beta

  • Remove Internet server port setting and always display URL
  • Enable internet connection by default
  • Properly free web socket connections
  • Properly cleanup zstd pipes
  • Move internet port setting into config file

Changes with server 2.5.10 beta

  • Implement connection establishment via web sockets
  • Fix file/zip download from partition mounted images
  • Pick up existing image mount path if current mount fails
  • Delete .bitmap file if present on image cleanup
  • Use navigator.languages (if present) to get date format method

Changes with client 2.5.6 beta

  • Use new settings handling for VSS component selection
  • Skip interfaces where address is not set to preven segfault
  • Update last modified time if settings use value gets updated
  • Use new settings handling in VSS component selection

Changes with client 2.5.5 beta

  • Adding settings status symbols to installers for Linux / macOS
  • Improve parallel hash logging
  • Disable mariadbxtrabackup (on Linux/macOS)
  • Fix cache target generation handling w.r.t parallel hashing

Changes with client 2.5.4 beta

  • Implement connection establishment via web sockets
  • Handle issue where directory doesn’t get iterated forward because of error
  • Use hole punching to set CBT hash data to zero
  • Escape backslash also if next character is [ in glob pattern

Downloads

Hello

What i did encounter was setting migrated but not applied properly.
With a client disallowed to change settings :
After the update , the config was set on se the setting from the client which was empty
Instead of use “setting configured here”.
Simply clicking on the icon once fixes the pb, just i don’t really know which settings have this issue.

Maybe also had an issue with the server key, but maybe it was on my side (different install path or whatever).

Apparently i got a pb with the scheduling in the latest version, a client keep doing the backups.
I am switching it to “setting configured here” to see if it get better.

Follow up regarding config icons on Client 2.5.4 beta:
The config icons are shown as long as they are found in the working directory (usually “C:\Program Files\UrBackup”). But when the Client gets launched via Registry-Run at startup (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run), the working directory is “C:\Windows\System32” and thus the icons won’t load. This is because “wxBitmap::LoadFile” searches the current working directory instead of the directory where the executable file is located.

I’m unsure which is the best way to solve this issue. Maybe “wxStandardPaths::GetResourcesDir” to point “wxBitmap::LoadFile” to the right directory? Or setting the working directory to the directory where the executable file is located?

Thanks, this should fix it

Thanks for the quick fix. There is another block starting at line 922 where res_path is missing:

Hello

The server stopped :confused: i got the debug symbols installed but it’s too late.

dpkg -l | grep urba
ii urbackup-server 2.5.10.0 amd64 Server for doing backups of clients
ii urbackup-server-dbg 2.5.10.0 amd64 debugging symbols for urbackup-server

2020-07-08 14:12:20: HT: Renaming file to “/data/urbackup2/xxx/200708-1358/.symlink_upath” with hash output
2020-07-08 14:12:21: Authed+capa for client ‘xxx’ (encrypted-v2, compressed-zstd, token auth) - 1 spare connections
2020-07-08 14:12:21: Connecting to target service…
2020-07-08 14:12:21: Established internet connection. Service=0
2020-07-08 14:12:22: Authed+capa for client ‘xxx’ (encrypted-v2, compressed-zstd, token auth) - 1 spare connections
2020-07-08 14:12:22: Connecting to target service…
2020-07-08 14:12:22: Established internet connection. Service=0
2020-07-08 14:12:22: Referencing snapshot on “xxx” for path “.symlink_vendor” failed: FAILED
2020-07-08 14:12:22: Loading file “.symlink_vendor” (metadata only)
2020-07-08 14:12:23: Authed+capa for client ‘xxx’ (encrypted-v2, compressed-zstd, token auth) - 1 spare connections
2020-07-08 14:12:23: Connecting to target service…
2020-07-08 14:12:23: Established internet connection. Service=0
2020-07-08 14:12:25: ERROR: Thread exit with unhandled std::exception OS_Rng: open /dev/urandom operation failed with error 24

ls -l /dev/urandom
crw-rw-rw- 1 root root 1, 9 Jul 7 14:43 /dev/urandom

uname -a
Linux xxx 5.4.0-40-generic #44-Ubuntu SMP Tue Jun 23 00:01:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

ulimit -Hn
1048576
ulimit -Sn
1024

I guess if you run it in gdb you could look into ls -lh /proc/PID/fd to see which files are open, i.e. what it leaks. Might even work with a server that doesn’t run as long yet.

It’s leaking sockets

ls -lh /proc/1074066/fd | grep socket |wc -l
14938

ls -lh /proc/1074066/fd | head -n20
total 0
lrwx------ 1 root root 64 Jul 9 12:41 0 -> /dev/null
lrwx------ 1 root root 64 Jul 9 12:41 1 -> /dev/null
lrwx------ 1 root root 64 Jul 9 12:41 10 -> /var/urbackup/backup_server.db-shm
lrwx------ 1 root root 64 Jul 9 12:41 100 -> /var/urbackup/backup_server_settings.db
lrwx------ 1 root root 64 Jul 9 12:41 1000 -> socket:[1573935]
lrwx------ 1 root root 64 Jul 9 12:41 10000 -> socket:[1620291]
lrwx------ 1 root root 64 Jul 9 12:41 10001 -> socket:[1620292]
lrwx------ 1 root root 64 Jul 9 12:41 10002 -> socket:[1614747]
lrwx------ 1 root root 64 Jul 9 12:41 10003 -> socket:[1620293]
lrwx------ 1 root root 64 Jul 9 12:41 10004 -> socket:[1614748]
lrwx------ 1 root root 64 Jul 9 12:41 10005 -> socket:[1614749]
lrwx------ 1 root root 64 Jul 9 12:41 10006 -> socket:[1614750]
lrwx------ 1 root root 64 Jul 9 12:41 10007 -> socket:[1614751]
lrwx------ 1 root root 64 Jul 9 12:41 10008 -> socket:[1620294]
lrwx------ 1 root root 64 Jul 9 12:41 10009 -> socket:[1620295]
lrwx------ 1 root root 64 Jul 9 12:41 1001 -> socket:[1573936]
lrwx------ 1 root root 64 Jul 9 12:41 10010 -> socket:[1620297]
lrwx------ 1 root root 64 Jul 9 12:41 10011 -> socket:[1620298]
lrwx------ 1 root root 64 Jul 9 12:41 10012 -> socket:[1620299]

I guess you don’t use the new web socket connection?

No, i only upgraded.

Trying to switch to websocket now. (gotta also pinpoint that scheduling thing)

Can i get websocket and tcp connections at the same time?
if i tell the server to listen to ws://127.0.0.1:55414/socket, how does the internet clients connect to update their settings and switch to websocket?
Or should i tell clients to use both type of connections, ie “127.0.0.1;ws://127.0.0.1:55414/socket” during the migration.

Also the port thing tickles me because in one scheme, the port is set inside the url, in the other outside. And if you use the websocket scheme, you still have to specify a port. I think it’s because one setting is a server setting (which port to listen), while the other setting is actually a client setting (which adresseS the clients needs to connect to).
Actually that goes for all settings in the “general” menu, some are the server settings (Max simultaneous backups), some are actually the default settings the server will use for a new group or client.

Also not reading properly i did a manual switch using
urbackupclientctl set-settings -k internet_server -v ws://xxx:55414/socket
And that does appear to work properly

but wss://xxx/socket gets me :
Trying to connect to internet server “wss://xxx/socket” at port 55414
Socket has error: 111
Connecting failed.

Yeah, the log message is wrong. Did you setup a web server with ssl (port 443) on the server?

Yes, clients connect via the “old” method, then get the new setting and eventually connect via that.

Yeah, that’s really confusing now. Needs rework.

No
Hoo, i think i understood
urbackupclientctl set-settings -k internet_server -v wss://62.210.182.207:55414/socket
Trying to connect to internet server “wss://xxx:55414/socket” at port 55414
WARNING: OpenSSL error: wrong version number
Connecting failed.

The error message is always about the same port, hence why you asked for 443
wss actualy stands for web socket secure, like http and https

So the good setting for me to only try to avoid leak is ws://xxx:55414/socket

Yeah, except I already found a socket leak in the web socket code :wink: . Perhaps you could track down which port the leaked sockets have? (if it is the http server or the internet clients)

Its from the urbackup client, it increase when you start a new backup, then get stuck in CLOSE_WAIT
TCP xxx:55414->zzz:39620 (CLOSE_WAIT)

lsof -p 2338562| grep CLOSE_WAIT |wc -l
1137

55414 is the http server port. If you are using web sockets now, I guess you can’t distinguish them anymore…

And CLOSE_WAIT, just keeps the port reserved for a time after close. You’d have wait a few minutes if linux removes them…

By removing websocket, we can see it comes from the client. But also the issue is compounded by having this scheduling issue where the backup keeps restarting (955 to 1885 fd).

urbackups 2344646 urbackup 86u IPv4 8622709 0t0 xxx:55415->yyy:39826 (CLOSE_WAIT)

date
Fri 10 Jul 2020 01:30:53 PM CEST

lsof -p 2344646 | grep CLOSE_WAIT |wc -l
953

date
Fri 10 Jul 2020 01:32:20 PM CEST

lsof -p 2344646 | grep CLOSE_WAIT |wc -l
954

date
Fri 10 Jul 2020 01:33:04 PM CEST
lsof -p 2344646 | grep CLOSE_WAIT |wc -l
955

ate
Fri 10 Jul 2020 01:33:42 PM CEST
lsof -p 2344646 | grep CLOSE_WAIT |wc -l
1885

date
Fri 10 Jul 2020 01:43:20 PM CEST
lsof -p 2344646 | grep CLOSE_WAIT |wc -l
1885

date
Fri 10 Jul 2020 01:44:01 PM CEST

Ok, so I think it is a problem with the ZSTD transfer compression. This was also present in 2.4.x, but no one reported it there? (or as high memory usage?)