Server 2.4.3 beta/Client 2.4.1 beta

uroni · July 1, 2019, 7:09pm

Major changes with server 2.4.x beta

IPv6 support (sponsored by Tuxis Internet Engineering )
ZSTD transport compression
Parallel hashing improvements

Changes with server 2.4.3 beta

Use heap for recursion when deleting non-empty directory
Make ipv6 optional per default

Changes with server 2.4.2 beta

Fix FreeBSD compile issue

Changes with server 2.4.2 beta

Fix FreeBSD compile issue

Changes with server 2.4.1 beta

Fix polling with ipv6 socket on Linux in ServiceAcceptor
Fix report script cache reload
Fixup backup folder location correctly (trailing slash)
Fix IPv6 on FreeBSD (not compile tested yet)

Changes with server 2.4.0 beta

Allow setting group id while adding new clients
Postpone metadata application of files that are in queue before reordered downloads (because of missing file in previous backup)
Allow ldap admin users access to all backups
Set file backup to incomplete during startup recovery
New setting to make backup dirs optional per default
Cleanup one image backup of each client at a time when freeing space (like with file backups)
Catch and log info about C++ exceptions
Improve parallel hash logging and properly set eof when finished
Do not retry getting parallel hash after error
Check if VHD file was opened successfully before using it (Fixes Windows crash)

Major changes with client 2.4.x beta

IPv6 support (sponsored by Tuxis Internet Engineering )
ZSTD transport compression
HTTPS proxy support (only with SChannel on Windows for now)
Parallel hashing improvements
Image backup performance improvement: Read-ahead file system in larger blocks

Changes with client 2.4.1 beta

Fix FreeBSD compile issue
Read-ahead file system in larger blocks during image backups
Make ipv6 optional per default
Return ipv6 zone in lookup and use it to connect

Changes with client 2.4.0 beta

Reset reconnect tries if more than 30min between reconnects
Only set reconnect_tries to 50 if it wasn’t set higher previously
Tool to read volume and write zeros to blocks which cannot be read
Set socket window size to 3MiB send buffer and 128KiB recv buffer for client on Windows
Fix incr image bit clashing with full file backup bit
Remove result writing via pipe fixing some async indexing hang-ups
Fix search for snapshot if VSS path is the root of a volume
Fix parallel hashing with VSS components
Hard code not following default wine z: symlink to root of fs on non-Windows
Functionality to interrupt parallel hash loading threads on clients
Don’t parallel hash files smaller than 2048 bytes
Add to db if there is a sub-directory because that one can be deleted and should then be removed from the db (Linux/MacOS)
Do not quit indexing with “no backup dirs” if components are selected for backup

Changes with Restore CD 2.2.2 beta

Fix GPT restore with restore disk being (slightly) smaller

Upgrade process

As always: Replace the executables (via the installers) and the database of the server/client will be updated on first running it.

Place the files from the update directory into C:\Program Files\UrBackupServer\urbackup or /var/urbackup to auto-update clients. Disable Download client from update server in the server settings to prevent the server from downloading the current version.

On Linux e.g. with this update script: https://github.com/ptempier/get_urbackupclient/blob/master/updateclient.sh

Downgrade process (server)

Stop the UrBackup server, restore C:\Program Files\UrBackupServer\urbackup or /var/urbackup from a backup before upgrade and then install the previous version over the beta release.

Downloads

Server: https://ssl.webpack.de/beta.urbackup.org/Server/2.4.3%20beta
Client: https://ssl.webpack.de/beta.urbackup.org/Client/2.4.1%20beta
Restore CD: https://www.urbackup.org/downloads/Restore/urbackup_restore_2.2.2.iso (amd64)

adumas · July 4, 2019, 1:03am

Server 2.4.3 compiles and works great on FreeNAS IPv4. Thanks!

karibbou · July 5, 2019, 12:23am

Hi

Any plans for i386 GNU/Linux builds?

Don_Wright · July 6, 2019, 3:02am

Since effectively zero Linux distros support plain i386 systems, I wouldn’t count on it. If you’re running one, you should try compiling from source and let us know how it works.

Bearded_Blunder · July 6, 2019, 9:39pm

I wouldn’t call Debian “effectively zero”, it’s been in the top 10 on distrowatch since forever. If you’d said 32 bit hardware was nearly extinct I’d be forced to agree, though I still have some in use.

Without UrBackup currently, I just take offline images using dd or similar & file backups using tar, I’m not expecting 32 bit debs to appear.

I’m assuming the OP meant 32 bit (plain x86, as opposed to x86_64)

Don_Wright · July 8, 2019, 1:34am

On a technical support forum or list I don’t assume the poster means anything other than what is said. No distro was mentioned, so I could only respond as if i386 means a system containing an Intel 80386 microprocessor. Whatever the processor actually in use, those unwilling to wait for a stable release when additional downloads are historically made available, may avail themselves of the beta .tar or other source openly provided.

Additional background, quoting from Debian’s notes regarding their current stable release in the flavor they call i386: (emphasis added)

2.1.2.1. CPU

Nearly all x86-based (IA-32) processors still in use in personal computers are supported. This also includes 32-bit AMD and VIA (former Cyrix) processors, and processors like the Athlon XP and Intel P4 Xeon.

However, Debian GNU/Linux buster will not run on 586 (Pentium) or earlier processors.

In case you missed it, Debian i386 hasn’t meant Intel i386, i486, or even Pentium (586) since 2016. The last release to support an Intel i486-series CPU is lenny from 2009. I can’t find any official notice of when i386 didn’t include Intel 80386; perhaps it died quietly somewhere in the 2.x releases. Intel stopped making them in 2007 and Linus Torvalds stopped supporting it in the kernel with 3.8.(Feb 2013.)

Jeeves · July 16, 2019, 10:00am

This doesn’t work for me in a ipv6-only environment.

2019-07-16 12:03:36: Trying to connect to internet server “urbackuptest.tuxis.net” at port 55415
2019-07-16 12:03:36: Socket has error: 111
2019-07-16 12:03:36: Connecting failed.

I do not see any packets to port 55415 from the client end.

uroni · July 16, 2019, 12:07pm

Could you run it in strace? (strace -f urbackupclientbackend -i -v debug ?

Jeeves · July 16, 2019, 1:33pm

975   socket(AF_INET6, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 12
975   connect(12, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "fc00:0:0:128::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0
975   poll([{fd=12, events=POLLOUT}], 1, 0) = 1 ([{fd=12, revents=POLLOUT}])
975   sendto(12, "\34\316\1\0\0\1\0\0\0\0\0\0\furbackuptest\5tuxis\3net\0\0\1\0\1", 40, MSG_NOSIGNAL, NULL, 0) = 40
975   poll([{fd=12, events=POLLIN}], 1, 5000) = 1 ([{fd=12, revents=POLLIN}])
975   ioctl(12, FIONREAD, [95])         = 0
975   recvfrom(12, "\34\316\201\203\0\1\0\0\0\1\0\0\furbackuptest\5tuxis\3net\0\0\1\0\1\300\31\0\6\0\1\0\0\0\235\0+\7nsauth1\300\31\nhostmaster\300\31xX\37\263\0\0*0\0\0\16\20\0\t:\200\0\0\16\20", 1024, 0, {sa_f
amily=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "fc00:0:0:128::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 95
975   close(12)                         = 0
975   socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 12
975   fcntl(12, F_GETFL)                = 0x2 (flags O_RDWR)
975   fcntl(12, F_SETFL, O_RDWR|O_NONBLOCK) = 0
975   connect(12, {sa_family=AF_INET, sin_port=htons(55415), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINPROGRESS (Operation now in progress)
975   poll([{fd=12, events=POLLOUT}], 1, 10000) = 1 ([{fd=12, revents=POLLOUT|POLLERR|POLLHUP}])
975   getsockopt(12, SOL_SOCKET, SO_ERROR, [ECONNREFUSED], [4]) = 0
975   close(12)                         = 0
975   stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2949, ...}) = 0
975   write(1, "2019-07-16 15:36:24: Socket has error: 111\n", 43) = 43
975   write(3, "2019-07-16 15:36:24: Socket has error: 111\n", 43) = 43
975   lseek(3, 0, SEEK_CUR)             = 17552
975   stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2949, ...}) = 0
975   write(1, "2019-07-16 15:36:24: Connecting failed.\n", 40) = 40
975   write(3, "2019-07-16 15:36:24: Connecting failed.\n", 40) = 40
975   lseek(3, 0, SEEK_CUR)             = 17592
975   futex(0x560c44592c70, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1563284484, tv_nsec=453738}, FUTEX_BITSET_MATCH_ANY <unfinished ...>

uroni · July 16, 2019, 5:17pm

I just tested it again on Linux and it seemed to work. It might be a problem with adding it to /etc/hosts. I couldn’t get it to work with link local addresses at all this way (/etc/hosts doesn’t allow adding the scope to ipv6 addresses). Plus it seems to skip looking at /etc/hosts… In the strace you can see it asking your dns server fc00:0:0:128::1 port 53 for the address and it returns ipv4 0.0.0.0 ?

I got it to work by adding a AAAA (only) record to a domain (would be urbackuptest.tuxis.net in your case guess).

Jeeves · July 17, 2019, 8:27am

So I did the following:

root@urbackupclient:/etc# cat /etc/resolv.conf 
#nameserver fc00:0:0:128::1
nameserver ::1
root@urbackupclient:/etc# dig @localhost urbackuptest.tuxis.net A

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> @localhost urbackuptest.tuxis.net A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43698
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;urbackuptest.tuxis.net.		IN	A

;; AUTHORITY SECTION:
tuxis.net.		3526	IN	SOA	nsauth1.tuxis.net. hostmaster.tuxis.net. 2019071703 10800 3600 604800 3600

;; Query time: 4 msec
;; SERVER: ::1#53(::1)
;; WHEN: Wed Jul 17 10:30:03 CEST 2019
;; MSG SIZE  rcvd: 106

root@urbackupclient:/etc# dig @localhost urbackuptest.tuxis.net AAAA

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> @localhost urbackuptest.tuxis.net AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34336
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;urbackuptest.tuxis.net.		IN	AAAA

;; ANSWER SECTION:
urbackuptest.tuxis.net.	3600	IN	AAAA	fc00::128:c4de:91ff:feed:259d

;; Query time: 5 msec
;; SERVER: ::1#53(::1)
;; WHEN: Wed Jul 17 10:30:06 CEST 2019
;; MSG SIZE  rcvd: 79

So I’ve setup a nameserver locally that can resolve all the required records. When starting urbackupclient, I only see a request for the A-record.

root@urbackupclient:/etc# 10:32:22.779985 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv6 (0x86dd), length 102: ::1.34234 > ::1.53: 63680+ A? urbackuptest.tuxis.net. (40)

No AAAA-records are requested.

Jeeves · July 17, 2019, 8:30am

The 0.0.0.0 is a choice Urbackup seems to make, btw. Dnsmasq returns ‘Refused’ and Powerdns returns ‘NxDomain’, which is better.

uroni · July 17, 2019, 12:41pm

Ok, following test code. Compile with:

curl https://gist.githubusercontent.com/uroni/adca370a15befc66898b5f93f78efed8/raw/d04ced53be884132712193ebb5f60666df2ffedf/test_lookup.cpp > test_lookup.cpp
g++ test_lookup.cpp -o test_lookup

I get e.g.:

app-server:~# ./test_lookup urbackup.org
Address (ipv6): 2604:a880:800:10::5e:e001
Address (ipv4): 162.243.187.177
app-server:~# ./test_lookup ipv6test.urbackup.org
Address (ipv6): 2604:a880:800:10::5e:e001

Jeeves · July 17, 2019, 1:14pm

This works!

root@urbackupclient:~# ./test_lookup urbackup.org
Address (ipv6): 2604:a880:800:10::5e:e001
Address (ipv4): 162.243.187.177
root@urbackupclient:~# ./test_lookup urbackupclienttest.tuxis.net
Address (ipv6): fc00::128:1097:63ff:fe20:1daa

Also, this honours the /etc/hosts entries. It does not contact the nameserver if an /etc/hosts entry is present!

uroni · July 17, 2019, 1:17pm

Hmm, do you use the binary client, and if yes, what kind of architecture does it install?

Jeeves · July 17, 2019, 1:43pm

I used both clients, the one from beta.urbackup.org and the one downloaded from the server.

I’m afraid to say so, but…

root@urbackupclient:~# urbackupclientctl status
{
"capability_bits": 65536,
"finished_processes": [],
"internet_connected": true,
"internet_status": "connected",
"last_backup_time": 0,
"running_processes": [{
"action": "FULL",
"eta_ms": 0,
"percent_done": -1,
"process_id": 1,
"server_status_id": 9,
"speed_bpms": 0
}
],
"servers": [{
"internet_connection": true,
"name": "urbackuptest.tuxis.net"
}
],
"time_since_last_lan_connection": 100275177
}

uroni · July 17, 2019, 1:47pm

So it works now, suddenly?

I was asking because the binary client uses different libcs depending on the system (it shows that during install). Either musl or glibc. I’ll change it from musl with the next build (to bionic). Musl and glibc use different lookup implementations and glibc is the build in one on most Linux systems.

Jeeves · July 17, 2019, 3:54pm

Yes, it works. I’m running on Buster, btw.

Jeeves · July 17, 2019, 3:58pm