I recently was having a problem where file backups on my Windows clients were consistently failing with indexing errors - for days. One of the things I tried to fix that was running the various batch files on the server to clean up the database, specifically running: cleanup.bat, cleanup_database.bat, and remove_unknown.bat. (Remove_unknown seemed to do the most from the messages displayed when it ran). None of these fixed the file backup indexing errors on the clients, but it seems like running them may have made one of my Linux clients permanently loose connection to the server.
The disconnected Linux client (ARM on a Raspberry Pi4) has perfect network connectivity to the server within the wi-fi local network, the “usbackupclientb” process is successfully running on the client and the client still shows up correctly in the server web site’s status and other displays. Shows up, but always disconnected now.
I tried re-installing the Linux client (successfully) and have rebooted the client system multiple times, but still that client continues to show up as disconnected from the server.
I thought about telling the server to remove the client, but have not done that because I’d have no idea how to get the server to find and re-connect to it again unless it would do that automatically.
Any thoughts or suggestions about what to try next would be greatly appreciated!
Look in the client’s log if there are any errors? You can also increase the verbosity of logging on the client to get more insight what’s not working.
Indexing errors during backup are likely a client-side issue, i.e. a directory that cannot be read, a dangling symlink, things like that. Typically on Windows you’ll get errors e.g. for OneDrive files that are currently not stored locally.
Thanks for your reply!
Forgot to mention that I was able to get past the Windows-client-file-backup-indexing-problems by changing the Default-Directories-to-Backup from using a wild card to a specific disk. I already excluded any non-local storage like OneDrive a long time ago, which i mostly don’t use anyway.
Good idea about checking the client logs but I’m not sure where to find them on Linux (or how to change any of the client parameters like log level locally) - since that client has no GUI and just quietly runs in the background. I usually set all its parameters from the server (not helpful when the client won’t connect to it), so never directly interact with it running on Linux at all. I’ll poke around in the documentation and on the Linux system to see if I can locate any logs that are useful.
I noticed that there is an “Add Client” button on the server’s status display, but I’ve never used it as all my local clients have always been discovered automatically in the past. Worried that if I try to delete the unconnected client in the server UI and then try to redefine it with the Add Client button that it may not take.
I really have no explanation as to why only one of my Linux clients suddenly became disconnected like this when all the other ones set up the same way have continued to work fine…
Thanks again for your suggestions!
Found the Linux client’s log file which is not named exactly as described in the admin manual (it is “/var/log/urbackupclient.log”) but doesn’t really show anything useful about why the client may not be connecting. Here’s the tail of it for the last day:
2023-03-02 03:18:48: WARNING: Shutting down (Signal 15)
2023-03-02 03:18:54: ERROR: Error joining ipv6 multicast group ff12::f894:d:dd00:ef91
2023-03-02 03:21:45: WARNING: Shutting down (Signal 15)
2023-03-02 03:21:52: ERROR: Error joining ipv6 multicast group ff12::f894:d:dd00:ef91
Everything in the local network is configured with IPv4, so I have no idea what the message about IPv6 has to do with anything. No log for that client created on the server side, since the last time the client did actually connect a few days ago.
So: The Urbackup client does at least seem to be running on the client system, the local network connectivity between the server and client systems is perfect (running real-time security camera displays), and as previously mentioned re-installing the Urbackup client software makes no difference.
The mystery continues…
I tried removing the constantly disconnected Linux client from the server’s web console and then waiting for it to auto-discover it (even giving it the IP address as a clue). No luck, it never finds it again. I even copied the server’s id and key (from the web console) to the server_idents.txt file on the client, but still no luck.
Next step will be to try to delete all the client parameter files and then re-install Urbackup once again - even though re-installing has not worked in the past. Maybe with the server_idents.txt and any other parameter files I see gone, it’ll work better and let the client get auto-discovered once again…
Deleted all the client parameter files I could find and then re-installed the client code - but still no luck, it never is discovered and never tries to connect to the local server. The “server_idents.txt” file remains the vanilla one without the correct server id and key for my local server.
During the client software re-install, it never asked me about image backups - which I seem to remember it did when I originally installed it for the first time. This implies that there may be other config files in a different place than the “server_idents.txt” file that need to be removed to really do a clean re-install. No idea how I’m going to find and remove those…
Spent an hour or so reading the administrator’s manual, looking for any clues. Section 4.3 has a discussion about client security, but didn’t explain to me how to run any client commands on Linux at all. It talked about the “pw.txt” and “pw_changed.txt” files being needed to do this, but given that they seem to contain hashed passwords (?) - still provided no details about how to use them to do anything.
Did another re-install of the whole Linux client installer package, but again never got the prompt asking me anything about what block image mechanism to use - leaving the apparently running “urbackupclientb” process still not connecting to the server at all. Again updated the “server_idents.txt” file to contain the server id and key from the “Add Client” screen on the server with no change, still no connection from the client.
Does anyone have any idea how to really accomplish a complete uninstall of the Urbackup client from a Linux system? I keep thinking that if I could do that (and completely removing the “/usr/local/var/urbackup” parameter directory with its sub-directories didn’t do it) that then a new re-install should hopefully allow to client to initialize itself to find and connect to the server once again.
Some progress, but still no solution to get the lost client reconnected to the local Urbackup server. I clobbered the “/usr/local/var/urbackup”, “/use/local/share/urbackup”, and “/use/local/etc/urbackup” directories on the client and then re-installed the client software again. This time I actually got the prompt asking me which block image mechanism I wanted - which had been missing from my previous re-install attempts. This didn’t fix the connectivity problem however (the client still never seems to contact the server at all), but it seems like a step forward at least…
On my local network, I have two Raspberry Pi Urbackup clients (both running the same Pi hardware (except one has 4GB of memory and the other has 8GB) and identical version of Ubuntu 22.04.1LTS). After both clients were working perfectly for several months, suddenly one system stopped being able to communicate with the server (always showing it was offline, even when it was perfectly connected via the network). This seemed to happen after I’d run several of the server database cleanup batch files, to fix indexing errors on some clients after I’d moved the backup data store to a new hard drive. The now offline Pi client wasn’t one of the systems having the indexing issues at all.
As detailed above, I’ve tried to re-install the Urbackup client software multiple times, often after totally removing the parameter directories, on the broken system without any success at getting it working again with my local Urbackup server (on Windows 11).
So last night I compared the various parameter files from both clients in “/usr/local/var/urbackup” and found:
-
The “server_idents.txt” files on both Pi systems seem to be exactly the same - containing the server’s id, finger print, and public key information. I don’t know how the broken, un-connecting Pi client system could have gotten this information except from contacting the local Urbackup server.
-
The working Pi system has a “tokens” sub-directory in “/usr/local/var/urbackup” that is missing from the non-working Pi’s configuration. This directory seems to have several files in it representing the various Linux users on the system that look like they contain some undefined cryptographic key?
-
On the working Pi system, the file “session_idents.txt” contains multiple lines, all with completely blank “secret_key=” fields on each line. In contrast, the non-working Pi system only had a single line in it with some daya defined in its “secret_key=” field. I tried clearing this data out and restarted the server, with no change to the non-connecting behavior.
-
Oh, as I’ve said several times in the previous posts the local network connectivity to both client systems is always stable and perfect. The working Pi system is hardwired to the local router while the non-working system (like most of my backup clients) is connected via 802.11ax wi-fi which always provides a perfect connection.
-
I’ve tried explictly adding the non-working Pi client into the server’s configuration using the “Add Client” button on the server’s status display screen. I found something weird about this last night. When I first try to give a client hint by entering its static IP address, it immediately shows up as “online” but then changes to not being online a moment or two later. However if I give the Pi client’s system name as a clue, it shows up as online indefinitely, even though it never successfully adds that client back into the systems being backed up.
-
From all of this, I conclude that the non-working client is running correctly and at some point after each time I’ve tried to re-install it is successfully connecting to my local server - even though the server is never accepting it as a new client once again.
-
As detailed in the posts above, I’ve also tried running the various batch files on the server to clean up and remove invalid content from the server’s databases - all with no positive effect.
-
A week or so ago, I tried explictly removing the client system from the server’s configuration (which was done) so that I could re-add it either automatically or explictly by using the “Add Client” button on the server’s status display screen. Since then, I’ve never gotten the client to show up again as a known client system.
So, is this a server configuration issue or something still not being right in the client’s configuration despite the repeated re-installs? If it is a server configuration issue, what can I do to get the server communicating with this client system once again?
The mystery continues but is getting tiresome at this point… Thanks for any further thoughts or suggestions!
This is what the “Add Client” hints screen has looked like for the past couple of days. “Online” but not connecting the client into the server’s configuration. The Linux client is running on the client, there is evedance that is has communicated with the server (to pick up its id and public key), and it’s been “online” - even though it still won’t add this client back into the server’s configuration. So what’s stopping it? What else do I need to try next to fix it?
It seems you have a similar issue I had. All my Linux clients were disconnected. Not the Windows ones.
Here my story All Linux clients offline - #4 by AnLAdmin
Complete uninstall and DB cleanup and waiting some days (I was on the verge to give up on urBackup). Suddenly they were connected again.
It also seems that disconnected clients doesn’t seem to prompt much interest in the forum. 
Hey, thanks for your response! Yeah, I’ve been getting tired of seeing my disconnected Linux client showing up as “online” in the Client Discovery Hints for nearly a couple of weeks now - without being accepted by the server again as a working client. I’ve been coming to the same conclusion that that is more likely a server problem than anything wrong with the disconnected Linux client.
I’ve tried all the various DB cleanup scripts a few times (especially to get around the separate problem of show stopper indexing errors on my Windows clients after I manually moved the server’s data repository to another drive) but they haven’t helped this issue so far. If I manually remove the server, I’m guessing that I’ll also have to manually remove and then reinstall all of the clients before I reinstall the server again - so that it will establish new client and server keys when the clients are discovered again. Makes it more complex and a bigger mess if the reinstalls of everything don’t get them to reconnect properly again afterwards.
So I guess the procedure I need to follow would be:
- Shutdown and manually remove the Windows server software and complete data repository.
- On each client, shut it down and manually remove the software and any associated configuration data (hoping that it’s not keeping stuff like the client and server keys hidden in the Windows registry where I probably couldn’t find them).
- With no clients then installed in the network, reinstall the server software and configure the server parameters - making sure to point the new data repository to the desired location.
- Reinstall each client and hope they initialize properly and connect with the server once again. (I’ll have to track down the laptops that my family uses which is easier said than done
). - Watch it recreate the backed up data repository again (which will take a day or two).
Can you (or anyone) think of any steps I’ve missed or put in the wrong order? This will be scary if I go through all this and it doesn’t work or makes things worse… Thanks again for your information and suggestions!
Not that I see. But probably a good idea to lookup additional infos in this forum regarding “moving” data store to a new server (similar to server reinstall with existing repository) and uninstall client. I thought I found one. There is even somewhere an uninstall script. Ah found it: Howto fully uninstall urbackup client on Ubuntu
You could also look into the db if all clients are removed before you try to add them. I think there are also some hints in the forum about that.
Would be nice if @uroni had some hints for situations like this. Because I never found anything in debug logs on server and client. I am still not sure why they suddenly connected.