Server 2.5.12 beta/Client 2.5.6 beta

uroni · July 6, 2020, 9:34pm

New settings handling

The major change with server 2.5.x and client 2.5.x is a complete redesign how settings are handled. The goal is to make it more explicit and consistent.

Each setting has a button next to it which switches between inheriting the setting from the group, using the configuration value on the configuration page and using the value from the client, and for some setting merging client and server settings together
File backup paths are now a normal setting just like any other. No more separation between paths to backup configured on the client and default directories to backup

This is a critical area of UrBackup and as such needs a lot of testing. Focus should be on if the 2.4.x client still more or less works with the 2.5.x server (and especially doesn’t loose its connection settings for Internet servers).

Connect via web sockets

With 2.5.x clients can connect via web sockets to the server. Setup should be easy. Instead of “Internet server name/IP” of e.g. “example.com” use “ws://example.com:55414/socket”. The real advantage comes into play if a real web server with SSL is setup in front. Either proxy the whole page, or serve the files and only proxy “/x” via FastCGI (see manual), additionally proxy “/socket” via web socket to “ws://127.0.0.1:55414/socket”. Then setup the clients to connect to “Internet server name” “wss://example.com/socket”. Basic authentication, a different name and a different port should also work, e.g. “wss://username:password@example.com/socket”.

Linux image backups

2.5.x can backup Linux clients (via dattobd only). The use case is e.g. backing up a DigitalOcean instance. It is explicitly not to create image backups of all possible configurations e.g. LVM, mdraid, btrfs, …
It can “live restore” a root volume, i.e., overwrite the root volume of a currently running machine. So one can image backup e.g. a DigitalOcean droplet. To restore create a new instance (tested with Debian 10/buster) then run the restore script which overwrites the current instance with the backed up instance using a trick to put the currently running system files into (compressed) memory.
Dattobd supports Changed Block Tracking (CBT), so image (and file) backups are done with CBT.

Linux image backup howto

Backup:

Install dattobd https://github.com/datto/dattobd/blob/master/INSTALL.md
To only backup used space install partclone (e.g. apt install partclone). This is currently supported for ext* and xfs. If partclone is not present it’ll backup unused space.
Install the (binary) client and select dattobd for snapshotting

Restore:

On the server browse to the image backup to restore
Click on “Restore Linux image”
Copy & paste the command into a new instance and follow the instructions (the command has a timeout (token), so after a while it should not work anymore)

Testing (image backup + file backup with CBT)

To test if CBT works correctly with file backups please enable Debugging: Verify file backups using client side hashes (for Internet clients with client-side hashing) or Debugging: End-to-end verification of all file backups in the advanced settings.

To test image backups enable md5sums creation on the client via touch /usr/local/var/create_md5sums_imagebackup. The client will then create a file (with date+time in the name) at /usr/local/var/md5sums-* for each image backup listing all files in the image plus their md5sum. Copy that file to server, mount the image backup then verify the md5sums with cd /backup/client/image_mnt; md5sums --quiet -c /path/to/md5sums-file.txt. The only files with checksum errors should be the .datto-*/.overlay-* files in the root directory and swap files (those get automatically excluded from backup).

Thanks for any help with testing and feedback!

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Place the files from the update directory into C:\Program Files\UrBackupServer\urbackup or /var/urbackup to auto-update clients. Disable Download client from update server in the server settings to prevent the server from downloading the current version.

On Linux e.g. with this update script: https://github.com/ptempier/get_urbackupclient/blob/master/updateclient.sh

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.

On testing beta/WIP versions

Please see Having problems with UrBackup? Please read before posting for what kind of information to add when reporting issues with the beta/WIP version. Reports such as “X does not work” are useless. In general, at the very least, think about what kind of information one would need to reproduce the problem in a different environment. Also a reminder that this is a community project which depends on your contribution and help. Testing is a way to contribute, but without detailed reports on the issues you are finding, you might as well wait for the final release. Best case you debug the problem yourself and then post the fix.

I’ll ignore any issue reports that don’t follow minimum reporting standards.

Changes with server 2.5.12 beta

Improve compatibility with older clients w.r.t. settings
Set current setting as client setting during database upgrade
Disable deletion of group archive entries
Only validate current settings when saving settings
Various new settings handling fixes
Add last modified time to each setting to resolve conflicts of which kind of setting should be used (from client or server)
For compatibility reasons add a setting for Internet port again and keep non-default setting on upgrade

Changes with server 2.5.11 beta

Remove Internet server port setting and always display URL
Enable internet connection by default
Properly free web socket connections
Properly cleanup zstd pipes
Move internet port setting into config file

Changes with server 2.5.10 beta

Implement connection establishment via web sockets
Fix file/zip download from partition mounted images
Pick up existing image mount path if current mount fails
Delete .bitmap file if present on image cleanup
Use navigator.languages (if present) to get date format method

Changes with client 2.5.6 beta

Use new settings handling for VSS component selection
Skip interfaces where address is not set to preven segfault
Update last modified time if settings use value gets updated
Use new settings handling in VSS component selection

Changes with client 2.5.5 beta

Adding settings status symbols to installers for Linux / macOS
Improve parallel hash logging
Disable mariadbxtrabackup (on Linux/macOS)
Fix cache target generation handling w.r.t parallel hashing

Changes with client 2.5.4 beta

Implement connection establishment via web sockets
Handle issue where directory doesn’t get iterated forward because of error
Use hole punching to set CBT hash data to zero
Escape backslash also if next character is [ in glob pattern

Downloads

Server: https://beta.urbackup.org/Server/2.5.12%20beta
Client: https://beta.urbackup.org/Client/2.5.6%20beta

orogor · July 7, 2020, 2:17pm

Hello

What i did encounter was setting migrated but not applied properly.
With a client disallowed to change settings :
After the update , the config was set on se the setting from the client which was empty
Instead of use “setting configured here”.
Simply clicking on the icon once fixes the pb, just i don’t really know which settings have this issue.

Maybe also had an issue with the server key, but maybe it was on my side (different install path or whatever).

orogor · July 8, 2020, 11:42am

Apparently i got a pb with the scheduling in the latest version, a client keep doing the backups.
I am switching it to “setting configured here” to see if it get better.

aliera · July 8, 2020, 1:53pm

Follow up regarding config icons on Client 2.5.4 beta:
The config icons are shown as long as they are found in the working directory (usually “C:\Program Files\UrBackup”). But when the Client gets launched via Registry-Run at startup (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run), the working directory is “C:\Windows\System32” and thus the icons won’t load. This is because “wxBitmap::LoadFile” searches the current working directory instead of the directory where the executable file is located.

github.com

uroni/urbackup_frontend_wx/blob/533113489709d73e10757f7925137b2d0e2f1fa0/gui/GUI.cpp#L137


	bSizer29->Add( m_textCtrl1, 0, wxALL, 5 );
	
	m_staticText2 = new wxStaticText( m_tab_filebackups, wxID_ANY, _("hours"), wxPoint( -1,-1 ), wxSize( -1,-1 ), 0 );
	m_staticText2->Wrap( -1 );
	bSizer29->Add( m_staticText2, 0, wxALIGN_CENTER|wxALL, 5 );
	
	wxSize image_button_size = wxDLG_UNIT(this, wxSize(15, 13));
	wxSize size_sub = wxDLG_UNIT(this, wxSize(5, 5));

	wxBitmap fa_home;
	fa_home.LoadFile("fa-home.png", wxBITMAP_TYPE_PNG);
	fa_home_img_scaled = fa_home.ConvertToImage().Scale(image_button_size.GetWidth()- size_sub.GetWidth(), image_button_size.GetHeight()- size_sub.GetHeight(), wxIMAGE_QUALITY_BILINEAR);

	wxBitmap fa_client;
	fa_client.LoadFile("fa-road.png", wxBITMAP_TYPE_PNG);
	fa_client_img_scaled = fa_client.ConvertToImage().Scale(image_button_size.GetWidth() - size_sub.GetWidth(), image_button_size.GetHeight() - size_sub.GetHeight(), wxIMAGE_QUALITY_BILINEAR);

	wxBitmap fa_lock;
	fa_lock.LoadFile("fa-lock.png", wxBITMAP_TYPE_PNG);
	fa_lock_img_scaled = fa_lock.ConvertToImage().Scale(image_button_size.GetWidth() - size_sub.GetWidth(), image_button_size.GetHeight() - size_sub.GetHeight(), wxIMAGE_QUALITY_BILINEAR);

I’m unsure which is the best way to solve this issue. Maybe “wxStandardPaths::GetResourcesDir” to point “wxBitmap::LoadFile” to the right directory? Or setting the working directory to the directory where the executable file is located?

uroni · July 8, 2020, 2:25pm

Thanks, this should fix it

aliera · July 8, 2020, 2:57pm

Thanks for the quick fix. There is another block starting at line 922 where res_path is missing:

github.com

uroni/urbackup_frontend_wx/blob/60467154d9713f622fcbd76a8046a33af60ec1fa/gui/GUI.cpp#L922


	bSizer32->Add( m_staticText27, 0, wxALIGN_CENTER|wxALL, 5 );
	
	m_textCtrl18 = new wxTextCtrl( this, wxID_ANY, wxEmptyString, wxDefaultPosition, wxDefaultSize, 0 );
	m_textCtrl18->Enable( false );
	m_textCtrl18->SetMinSize(wxDLG_UNIT(this, wxSize( 100,-1 )) );

	wxSize image_button_size = wxDLG_UNIT(this, wxSize(15, 13));
	wxSize size_sub = wxDLG_UNIT(this, wxSize(5, 5));

	wxBitmap fa_home;
	fa_home.LoadFile("fa-home.png", wxBITMAP_TYPE_PNG);
	fa_home_img_scaled = fa_home.ConvertToImage().Scale(image_button_size.GetWidth() - size_sub.GetWidth(), image_button_size.GetHeight() - size_sub.GetHeight(), wxIMAGE_QUALITY_BILINEAR);

	wxBitmap fa_client;
	fa_client.LoadFile("fa-road.png", wxBITMAP_TYPE_PNG);
	fa_client_img_scaled = fa_client.ConvertToImage().Scale(image_button_size.GetWidth() - size_sub.GetWidth(), image_button_size.GetHeight() - size_sub.GetHeight(), wxIMAGE_QUALITY_BILINEAR);

	wxBitmap fa_lock;
	fa_lock.LoadFile("fa-lock.png", wxBITMAP_TYPE_PNG);
	fa_lock_img_scaled = fa_lock.ConvertToImage().Scale(image_button_size.GetWidth() - size_sub.GetWidth(), image_button_size.GetHeight() - size_sub.GetHeight(), wxIMAGE_QUALITY_BILINEAR);

orogor · July 9, 2020, 8:24am

Hello

The server stopped i got the debug symbols installed but it’s too late.

dpkg -l | grep urba
ii urbackup-server 2.5.10.0 amd64 Server for doing backups of clients
ii urbackup-server-dbg 2.5.10.0 amd64 debugging symbols for urbackup-server

2020-07-08 14:12:20: HT: Renaming file to “/data/urbackup2/xxx/200708-1358/.symlink_upath” with hash output
2020-07-08 14:12:21: Authed+capa for client ‘xxx’ (encrypted-v2, compressed-zstd, token auth) - 1 spare connections
2020-07-08 14:12:21: Connecting to target service…
2020-07-08 14:12:21: Established internet connection. Service=0
2020-07-08 14:12:22: Authed+capa for client ‘xxx’ (encrypted-v2, compressed-zstd, token auth) - 1 spare connections
2020-07-08 14:12:22: Connecting to target service…
2020-07-08 14:12:22: Established internet connection. Service=0
2020-07-08 14:12:22: Referencing snapshot on “xxx” for path “.symlink_vendor” failed: FAILED
2020-07-08 14:12:22: Loading file “.symlink_vendor” (metadata only)
2020-07-08 14:12:23: Authed+capa for client ‘xxx’ (encrypted-v2, compressed-zstd, token auth) - 1 spare connections
2020-07-08 14:12:23: Connecting to target service…
2020-07-08 14:12:23: Established internet connection. Service=0
2020-07-08 14:12:25: ERROR: Thread exit with unhandled std::exception OS_Rng: open /dev/urandom operation failed with error 24

ls -l /dev/urandom
crw-rw-rw- 1 root root 1, 9 Jul 7 14:43 /dev/urandom

uname -a
Linux xxx 5.4.0-40-generic #44-Ubuntu SMP Tue Jun 23 00:01:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

ulimit -Hn
1048576
ulimit -Sn
1024

uroni · July 9, 2020, 8:50am

I guess if you run it in gdb you could look into ls -lh /proc/PID/fd to see which files are open, i.e. what it leaks. Might even work with a server that doesn’t run as long yet.

orogor · July 9, 2020, 10:43am

It’s leaking sockets

ls -lh /proc/1074066/fd | grep socket |wc -l
14938

ls -lh /proc/1074066/fd | head -n20
total 0
lrwx------ 1 root root 64 Jul 9 12:41 0 -> /dev/null
lrwx------ 1 root root 64 Jul 9 12:41 1 -> /dev/null
lrwx------ 1 root root 64 Jul 9 12:41 10 -> /var/urbackup/backup_server.db-shm
lrwx------ 1 root root 64 Jul 9 12:41 100 -> /var/urbackup/backup_server_settings.db
lrwx------ 1 root root 64 Jul 9 12:41 1000 -> socket:[1573935]
lrwx------ 1 root root 64 Jul 9 12:41 10000 -> socket:[1620291]
lrwx------ 1 root root 64 Jul 9 12:41 10001 -> socket:[1620292]
lrwx------ 1 root root 64 Jul 9 12:41 10002 -> socket:[1614747]
lrwx------ 1 root root 64 Jul 9 12:41 10003 -> socket:[1620293]
lrwx------ 1 root root 64 Jul 9 12:41 10004 -> socket:[1614748]
lrwx------ 1 root root 64 Jul 9 12:41 10005 -> socket:[1614749]
lrwx------ 1 root root 64 Jul 9 12:41 10006 -> socket:[1614750]
lrwx------ 1 root root 64 Jul 9 12:41 10007 -> socket:[1614751]
lrwx------ 1 root root 64 Jul 9 12:41 10008 -> socket:[1620294]
lrwx------ 1 root root 64 Jul 9 12:41 10009 -> socket:[1620295]
lrwx------ 1 root root 64 Jul 9 12:41 1001 -> socket:[1573936]
lrwx------ 1 root root 64 Jul 9 12:41 10010 -> socket:[1620297]
lrwx------ 1 root root 64 Jul 9 12:41 10011 -> socket:[1620298]
lrwx------ 1 root root 64 Jul 9 12:41 10012 -> socket:[1620299]

uroni · July 9, 2020, 10:44am

I guess you don’t use the new web socket connection?

orogor · July 9, 2020, 12:24pm

No, i only upgraded.

Trying to switch to websocket now. (gotta also pinpoint that scheduling thing)

orogor · July 10, 2020, 9:58am

Can i get websocket and tcp connections at the same time?
if i tell the server to listen to ws://127.0.0.1:55414/socket, how does the internet clients connect to update their settings and switch to websocket?
Or should i tell clients to use both type of connections, ie “127.0.0.1;ws://127.0.0.1:55414/socket” during the migration.

Also the port thing tickles me because in one scheme, the port is set inside the url, in the other outside. And if you use the websocket scheme, you still have to specify a port. I think it’s because one setting is a server setting (which port to listen), while the other setting is actually a client setting (which adresseS the clients needs to connect to).
Actually that goes for all settings in the “general” menu, some are the server settings (Max simultaneous backups), some are actually the default settings the server will use for a new group or client.

Also not reading properly i did a manual switch using
urbackupclientctl set-settings -k internet_server -v ws://xxx:55414/socket
And that does appear to work properly

but wss://xxx/socket gets me :
Trying to connect to internet server “wss://xxx/socket” at port 55414
Socket has error: 111
Connecting failed.

uroni · July 10, 2020, 10:28am

Yeah, the log message is wrong. Did you setup a web server with ssl (port 443) on the server?

Yes, clients connect via the “old” method, then get the new setting and eventually connect via that.

Yeah, that’s really confusing now. Needs rework.

orogor · July 10, 2020, 10:37am

No
Hoo, i think i understood
urbackupclientctl set-settings -k internet_server -v wss://62.210.182.207:55414/socket
Trying to connect to internet server “wss://xxx:55414/socket” at port 55414
WARNING: OpenSSL error: wrong version number
Connecting failed.

The error message is always about the same port, hence why you asked for 443
wss actualy stands for web socket secure, like http and https

So the good setting for me to only try to avoid leak is ws://xxx:55414/socket

uroni · July 10, 2020, 10:54am

Yeah, except I already found a socket leak in the web socket code . Perhaps you could track down which port the leaked sockets have? (if it is the http server or the internet clients)

orogor · July 10, 2020, 11:16am

Its from the urbackup client, it increase when you start a new backup, then get stuck in CLOSE_WAIT
TCP xxx:55414->zzz:39620 (CLOSE_WAIT)

lsof -p 2338562| grep CLOSE_WAIT |wc -l
1137

uroni · July 10, 2020, 11:22am

55414 is the http server port. If you are using web sockets now, I guess you can’t distinguish them anymore…

And CLOSE_WAIT, just keeps the port reserved for a time after close. You’d have wait a few minutes if linux removes them…

orogor · July 10, 2020, 11:46am

By removing websocket, we can see it comes from the client. But also the issue is compounded by having this scheduling issue where the backup keeps restarting (955 to 1885 fd).

urbackups 2344646 urbackup 86u IPv4 8622709 0t0 xxx:55415->yyy:39826 (CLOSE_WAIT)

date
Fri 10 Jul 2020 01:30:53 PM CEST

lsof -p 2344646 | grep CLOSE_WAIT |wc -l
953

date
Fri 10 Jul 2020 01:32:20 PM CEST

lsof -p 2344646 | grep CLOSE_WAIT |wc -l
954

date
Fri 10 Jul 2020 01:33:04 PM CEST
lsof -p 2344646 | grep CLOSE_WAIT |wc -l
955

ate
Fri 10 Jul 2020 01:33:42 PM CEST
lsof -p 2344646 | grep CLOSE_WAIT |wc -l
1885

date
Fri 10 Jul 2020 01:43:20 PM CEST
lsof -p 2344646 | grep CLOSE_WAIT |wc -l
1885

date
Fri 10 Jul 2020 01:44:01 PM CEST

uroni · July 10, 2020, 1:43pm

Ok, so I think it is a problem with the ZSTD transfer compression. This was also present in 2.4.x, but no one reported it there? (or as high memory usage?)