Sadness after server 2.4.9/client 2.4.8

Last Sunday, I updated my home UrBackup server to 2.4.9 (on Debian Buster). Since then, most of my clients (all Linux, mostly Debian) have stopped working. I’ve just updated to 2.4.10 about 30 minutes ago as I try to solve the problem, but this didn’t help.

The clients that did remain working are all still running older versions of the client - 2.3.4 and 2.1.15. All the ones listed as 2.4.8 are failed.
Also notable is that all the failed 2.4.8 clients are showing up under different names in the server… Previously, most clients were listed as their FQDN (which I prefer), but the failed ones have all been duplicated. They now show up as a short hostname instead, with a different backup set for each entry. Yes, both come up as offline.

I have got one of the clients to start working again. After updating the server to 2.4.10, I found an old 2.2.6 installer on the client I was fiddling with. Since it was a client-specific installer downloaded from my server for that client, I figured it couldn’t hurt, and installed it… This allowed the client to reconnect (using it’s old FQDN name) and start a backup. However, the client then suddenly went offline again.
It turns out the client updated itself to 2.4.8 again. I had turned off automatic updates, so this was a little surprising, however I manually restarted the client (it failed to start after the update), and it appears to be working OK again.

I guess my question now is… What do I do with the other 15 or so failed clients? Should I go through one by one downloading the custom installer for each client?

Looks like we are having the same issue!

Same here… was working with server 2.4.8, but 2.4.10 server refuses to connect to the clients.

Adding to this again - the client that re-updated to 2.4.8 disappeared again shortly after my initial post.
I have now manually downloaded client update files for 2.4.9 and created the install package for my test client, which failed in even more spectacular fashion:
root@bob:~# sh UrBackup\ Client\ (bob.home)-2.4.9.sh
Verifying archive integrity… All good.
Uncompressing UrBackup Client Installer for Linux 100%
Installation of UrBackup Client 2.4.9 to /usr/local … Proceed ? [Y/n]

Uncompressing install data...
Detected Debian (derivative) system
Detected architecture i686-linux-android
Illegal instruction
Error running executable on this system (i686). Stopping installation.

What’s the go with it listing android? And why does it think 32 bit is a no-go? Has 32 bit been dropped or is this a broken platform detection script?

However, manually downloading the client update files for 2.3.4 has got me going again. Something has definitely been broken in the Linux client between 2.3.4 and 2.4.8.

See here on how to debug the illegal instruction problem: https://forums.urbackup.org/t/ubuntu-glibc-not-installed-or-too-old/

i.e.

wget “https://hndl.urbackup.org/Client/2.4.9/UrBackup%20Client%20Linux%202.4.9-dbg.sh
sh '“UrBackup Client Linux 2.4.9-dbg.sh” --target /tmp/urbackup_installer --noexec
tar xf /tmp/urbackup_installer/install-data.tar.gz -C /tmp/urbackup_installer/
gdb --args /tmp/urbackup_installer/i686-linux-android/urbackupclientbackend --version
run
bt
x/i $pc

Also please post the output of cat /proc/cpuinfo.

Thanks!

This is running on Debian Stretch in an LXC container (on proxmox). It’s a 32 bit container on 64 bit host.

CPU info:

processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 8
model name : AMD Opteron™ Processor 4130
stepping : 0
microcode : 0x10000da
cpu MHz : 2599.754
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save pausefilter
bugs : tlb_mmatch fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2
bogomips : 5199.50
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

And the trace:

GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “i686-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from /tmp/urbackup_installer/i686-linux-android/urbackupclientbackend…done.
run
(gdb) run
Starting program: /tmp/urbackup_installer/i686-linux-android/urbackupclientbackend --version

Program received signal SIGILL, Illegal instruction.
0x08640dbf in chacha_encrypt_bytes (x=0xf7ff6008, m=0xf7ff6048 “”, c=0xf7ff6048 “”, bytes=1024) at bionic/libc/upstream-openbsd/lib/libc/crypt/chacha_private.h:143
143 bionic/libc/upstream-openbsd/lib/libc/crypt/chacha_private.h: No such file or directory.
(gdb) bt
#0 0x08640dbf in chacha_encrypt_bytes (x=0xf7ff6008, m=0xf7ff6048 “”, c=0xf7ff6048 “”, bytes=1024) at bionic/libc/upstream-openbsd/lib/libc/crypt/chacha_private.h:143
#1 _rs_rekey (dat=, datlen=) at bionic/libc/upstream-openbsd/lib/libc/crypt/arc4random.c:125
#2 0x08640a49 in _rs_random_buf (_buf=, n=4) at bionic/libc/upstream-openbsd/lib/libc/crypt/arc4random.c:161
#3 arc4random_buf (buf=0x8811ebc <__stack_chk_guard>, n=4) at bionic/libc/upstream-openbsd/lib/libc/crypt/arc4random.c:195
#4 0x08692efd in __libc_safe_arc4random_buf (buf=0x8811ebc <__stack_chk_guard>, n=4) at bionic/libc/bionic/bionic_arc4random.cpp:44
#5 0x086433ac in __libc_init_main_thread_late () at bionic/libc/bionic/__libc_init_main_thread.cpp:104
#6 0x086257f2 in __real_libc_init (raw_args=, onexit=, slingshot=, structors=0xffffdcd8, temp_tcb=0xffffdc80) at bionic/libc/bionic/libc_init_static.cpp:178

Thanks! Could you run the x/i $pc command in gdb as well, like I already mentioned?

Never mind. I can break at that point. It’s

(gdb) x/i $pc
=> 0x8640dbf <_rs_rekey+415>: pshufb 0x20(%esp),%xmm3

Sp pshufb which seems to be missing in some older AMD processors.

Oh… So does that mean I have to stick with older clients until I replace the server?

Could you try if this one works? http://beta.urbackup.org/tmp/UrBackupUpdateLinux-dbg(1).sh

Finally got around to checking that - seems to work fine on both 32 and 64 bit containers on the AMD server. :+1:

Edit: Correction - the 32 bit one is still running the client, but it’s now showing offline in the web GUI, and seems to not be doing anything. The server is doing a nightly cleanup though, so could that stop it showing up if it’s done the “I’m a new client” thing?