Server 2.5.5 WIP/Client 2.5.3 WIP

Testing
,
1

New settings handling

The major change with server 2.5.x and client 2.5.x is a complete redesign how settings are handled. The goal is to make it more explicit and consistent.

  • Each setting has a button next to it which switches between inheriting the setting from the group, using the configuration value on the configuration page and using the value from the client, and for some setting merging client and server settings together
  • File backup paths are now a normal setting just like any other. No more separation between paths to backup configured on the client and default directories to backup

This is a critical area of UrBackup and as such needs a lot of testing. Focus should be on if the 2.4.x client still more or less works with the 2.5.x server (and especially doesn’t loose its connection settings for Internet servers).

Linux image backups

2.5.x can backup Linux clients (via dattobd only). The use case is e.g. backing up a DigitalOcean instance. It is explicitly not to create image backups of all possible configurations e.g. LVM, mdraid, btrfs, …
It can “live restore” a root volume, i.e., overwrite the root volume of a currently running machine. So one can image backup e.g. a DigitalOcean droplet. To restore create a new instance (tested with Debian 10/buster) then run the restore script which overwrites the current instance with the backed up instance using a trick to put the currently running system files into (compressed) memory.
Dattobd supports Changed Block Tracking (CBT), so image (and file) backups are done with CBT.

Linux image backup howto

Backup:

  • Install dattobd https://github.com/datto/dattobd/blob/master/INSTALL.md
  • To only backup used space install partclone (e.g. apt install partclone). This is currently supported for ext* and xfs. If partclone is not present it’ll backup unused space.
  • Install the (binary) client and select dattobd for snapshotting

Restore:

  • On the server browse to the image backup to restore
  • Click on “Restore Linux image”
  • Copy & paste the command into a new instance and follow the instructions (the command has a timeout (token), so after a while it should not work anymore)

Testing (image backup + file backup with CBT)

To test if CBT works correctly with file backups please enable Debugging: Verify file backups using client side hashes (for Internet clients with client-side hashing) or Debugging: End-to-end verification of all file backups in the advanced settings.

To test image backups enable md5sums creation on the client via touch /usr/local/var/create_md5sums_imagebackup. The client will then create a file (with date+time in the name) at /usr/local/var/md5sums-* for each image backup listing all files in the image plus their md5sum. Copy that file to server, mount the image backup then verify the md5sums with cd /backup/client/image_mnt; md5sums --quiet -c /path/to/md5sums-file.txt. The only files with checksum errors should be the .datto-*/.overlay-* files in the root directory and swap files (those get automatically excluded from backup).

Thanks for any help with testing and feedback!

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Place the files from the update directory into C:\Program Files\UrBackupServer\urbackup or /var/urbackup to auto-update clients. Disable Download client from update server in the server settings to prevent the server from downloading the current version.

On Linux e.g. with this update script: https://github.com/ptempier/get_urbackupclient/blob/master/updateclient.sh

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.

On testing beta/WIP versions

Please see Having problems with UrBackup? Please read before posting for what kind of information to add when reporting issues with the beta/WIP version. Reports such as “X does not work” are useless. In general, at the very least, think about what kind of information one would need to reproduce the problem in a different environment. Also a reminder that this is a community project which depends on your contribution and help. Testing is a way to contribute, but without detailed reports on the issues you are finding, you might as well wait for the final release. Best case you debug the problem yourself and then post the fix.

I’ll ignore any issue reports that don’t follow minimum reporting standards.

Changes with server 2.5.5 beta

  • Fix blockalign reading/writing to stdin/stdout
  • Make last modification time configurable from script list
  • Fix backup of symlinks to directories with special characters

Changes with server 2.5.4 beta

  • Update SQLite
  • Fix JavaScript issue with settings
  • Fix db recovery (db repair script)

Changes with server 2.5.3 beta

  • Fix db upgrade issue

Changes with server 2.5.2 beta

  • Increase timeouts during backup starts from 10s to 60s
  • Multi-threaded image compression + compress image with ZSTD now
  • Forward image capa to alert script and use it to switch instead of os info
  • Backup and restore grub stage 1.5
  • Make cleanup log message less ambiguous
  • Check if client name is correct before starting backup and offline client if not
  • Upgrade LMDB to version 0.9.24
  • Use MDB_WRITEMAP and disable MDB_NOMETASYNC
  • Don’t encrypt server restore identity when starting restore from web interface
  • Fix some issues with the web interface for the new settings handling
  • Reset r_online when exiting reconnect loop in image backup

Changes with server 2.5.1 beta

  • Changes to support (limited) Linux image backups
  • Fixes to new settings handling
  • Upgrade to Visual Studio 2019
  • Use getrandom instead of reading from /dev/urandom if possible
  • Return error message if authentication key is empty
  • Fix check if image backup is already queued
  • Handle “all_nonusb” setting in image snapshot group
  • Fix storing alert state when serialization has null bytes
  • Fix backup_dirs_optional display + saving
  • Substract sparse extents before checking if file fits on storage
  • Upgrade miniz
  • Fix polish language name
  • Setting to only allow access to settings after the user enters a text
  • Add option to send mail SSL encrypted (without STARTTLS)

Changes with client 2.5.3 beta

  • Enable script backup for sub-clients
  • Fix mariadb backup by using mbstream format
  • Ignore backup GPT errors and use primary twice instead on error
  • Fix “ALL” file backup dirs handling

Changes with client 2.5.2 beta

  • Remove outdated log message for token not found
  • Fix problem in fix for piped tar file hash issue after resume
  • Check if client name is correct before starting backup and offline client if not
  • Don’t encrypt server restore identity when starting restore from web interface
  • Improve killing processes with files open for write when restoring to root device on Linux

Changes with client 2.5.1 beta

  • Changes to support (limited) Linux image backups
  • Upgrade to Visual Studio 2019
  • Use getrandom instead of reading from /dev/urandom if possible
  • Fix rare race condition that makes indexing hange after reconnect

Changes with server 2.5.0 beta

  • Rework settings handling

Changes with client 2.5.0 beta

  • Rework settings handling

Downloads

1 Like
3

Hello! I would like to try it, but I can’t download it. The links don’t work.

4

Just checked. Works for me.

5

I tried through Tor Browser, everything works fine.
But it still does not work from my IP.

Probably your server filtered my IP.

6

There is no filtering on the VM level at least… Can you do a traceroute beta.urbackup.org ? The server is involved in the server client download/update, so it would be interesting to know why this doesn’t work.

7

I thought may be the problem in my home ISP, but it’s not.
I checked it from my mobile thought mobile internet, it still doesn’t work.
I checked it from RDP server at datacenter, it still doesn’t work.

At my home ISP:

# traceroute beta.urbackup.org
traceroute to beta.urbackup.org (46.101.244.189), 30 hops max, 60 byte packets
 1  10.10.10.1 (10.10.10.1)  0.265 ms  0.348 ms  0.420 ms
 2  192.168.91.1 (192.168.91.1)  1.277 ms  1.340 ms  1.890 ms
 3  95-165-128-1.static.spd-mgts.ru (95.165.128.1)  6.183 ms  6.282 ms  6.415 ms
 4  10.109.12.125 (10.109.12.125)  7.724 ms  7.837 ms  8.139 ms
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

At RDP server at datacenter (CMD):

tracert beta.urbackup.org

Route trace to update.urbackup.org [46.101.244.189]
with a maximum number of jumps of 30:

   1 <1 ms <1 ms <1 ms 172.31.162.6
   2 1 ms <1 ms <1 ms 172.31.163.217
   3 <1 ms <1 ms <1 ms 172.17.23.72
   4 <1 ms <1 ms <1 ms zapret.hoztnode.net [92.63.108.22]
   5 zapret.hoztnode.net [92.63.108.22] reports: The specified node is unavailable.

Trace completed.

The server have same problem with update.urbackup.org:

2020-04-21 03:17:26: ERROR: Error while downloading version info from http://update.urbackup.org/2.4.x/version.txt: Couldn't connect to server(ec=7), Failed to connect to update.urbackup.org port 80: Connection timed out
2020-04-21 03:17:58: ERROR: Error while downloading version info from http://update.urbackup.org/2.4.x/version_linux.txt: Couldn't connect to server(ec=7), Failed to connect to update.urbackup.org port 80: Connection timed out

But in same time urbackup.org works fine:
At my home ISP (urbackup.org):

# traceroute urbackup.org
traceroute to urbackup.org (162.243.187.177), 30 hops max, 60 byte packets
 1  10.10.10.1 (10.10.10.1)  0.249 ms  0.339 ms  0.404 ms
 2  192.168.91.1 (192.168.91.1)  1.233 ms  1.284 ms  1.519 ms
 3  95-165-128-1.static.spd-mgts.ru (95.165.128.1)  5.991 ms  6.070 ms  6.193 ms
 4  mpts-a197-51.msk.mts-internet.net (212.188.1.106)  7.262 ms  7.501 ms  7.648 ms
 5  a197-cr04-be12.51.msk.mts-internet.net (212.188.1.105)  7.993 ms  8.115 ms  9.135 ms
 6  anc-cr01-be6.77.ff.mts-internet.net (212.188.2.189)  43.648 ms  38.934 ms  38.995 ms
 7  mil-cr01-be5.204.lnd.mts-internet.net (195.34.53.226)  136.428 ms *  136.559 ms
 8  sd-cr01-te2-3.190.nyc.mts-internet.net (195.34.53.142)  145.956 ms  146.109 ms  146.240 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

At RDP server at datacenter (CMD) (urbackup.org):

> tracert urbackup.org

Tracing the route to urbackup.org [162.243.187.177]
with a maximum number of jumps of 30:

   1 <1 ms <1 ms <1 ms 172.31.162.6
   2 1 ms <1 ms <1 ms 172.31.163.217
   3 3 ms <1 ms <1 ms 172.17.23.72
   4 <1 ms <1 ms <1 ms edge.webdc.ru [92.63.108.97]
   5 1 ms 1 ms 1 ms 89.22.16.154
   6 1 ms 1 ms 1 ms ae11-277.RT.MR.MSK.RU.retn.net [87.245.253.1]
   7 111 ms 112 ms 110 ms ae0-8.RT.TLX.NYC.US.retn.net [87.245.233.114]
   8 121 ms 121 ms 121 ms UnAssigned170.nyiix.net [198.32.160.170]
   9 116 ms 119 ms 116 ms 138.197.244.39
  10 * * * Timed out waiting for request.
  11 118 ms 118 ms 119 ms 162.243.187.177

Trace completed.
8

You pmed me the IP, so it’s the same as this thread Error auto update 2.4.12 (your government’s fault)

9

Yes you right!

The IP address 46.101.244.189 was blocked (not domain beta.urbackup.org).
This block affects to this subnet 46.101.128.0/17 (not one IP).

I read the reason for blocking this IP address. It got into blacklist because of telegram app.
Telegram app is blocked in Russia. And all IP addresses that the telegram service uses are also blacklisted.

May be at your IP address 46.101.244.189 there are another hosts which are telegram mirrors.
Or maybe you started using this address recently, and the previous owners used it as a telegram mirror, so it was blocked earlier.

About exclude IP address from blocking:
It is impossible to exclude an IP address from blocking, except to go to court.
This is the only option so far - to use a special browser (like a Tor) and hope that someday you will change the IP address of the host.

10

They are blocking random DigitalOcean/AWS IP-ranges and Telegram is still working in Russia… so fail. See Blocking of Telegram in Russia - Wikipedia

The technical process of this restriction began on April 16, 2018.[1] The blocking led to interruptions in the operation of many third-party services, but practically did not affect the availability of Telegram in Russia.

In April 2020, the Government of Russia started using the blocked Telegram platform to spread information related to COVID-19 outbreak.

Now all this is a bit off-topic, so let’s drop this…

11

I upgraded to this beta: Windows client 2.5.3. FreeBSD (FreeNAS) Server 2.5.5.
Since then I have issues with Images backups schedule. There seems to be a clash between client settings for frequency to images backups and server setting for those clients.

Right now clients keep making Full image backups on a continuous basis right after finishing one even though full backup is set @ 30 days.

I tried downgrading to server 2.4.12 but database is not compatible it seems.

On the status page I see a backup was made today but status says : No recent backup.

12

hi, is there any posibility that Server 2.5.5 beta make it compatible with debian buster? it can not be installed because libc6 version on buster is 2.28.

i know there’s ubuntu 20.04 LTS just released but i prefer non bloated distros.

thanks for all your work.

13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.