Urbackup server persistently crashes

Server
1

System

Raspberry Pi 4 4 Gb RAM, 32 Gb SD card running Raspberry Pi OS 11 (Bullseye)

Externally connected array of 4 x 2 Tb hard disks via USB 3 configured as RAID 10

Both Raspberry Pi and disk array are powered via mains electricity filter and UPS

Background

I have been successfully running UrBackup since June 2020, upgrading server and client software as new releases were made available.

A nearby thunderstorm a couple of weeks ago knocked out the power. Power was not restored by the time UPS exhausted. When power returned Raspberry Pi had powered up but disk array remained in standby which caused the next backup to be written to the SD card until it was full.

I tried to recover the mess and in the end gave up and decided to start from scratch.

Next Steps

Installed Raspberry Pi OS 11 on brand new 32 Gb SD card.

Decided to switch the external RAID 10 array from ext4 to btrfs.

Tested that btrfs was working.

Modified fstab to automount external array at required location

Rebooted to check that external array successfully automounted at required location

All fully functional

Made rsync backup of selected directories of Synology NAS using Hyperbackup.

130 Gb backup successful

Installed Urbackup server from repository following instructions on website

Install successful

Noticed that Urbackup kept on crashing (ps ax | grep urb returned no process)

Configured Urbackup as per prompts and modified /etc/default/urbackupsrv log level to “debug”

Restarted Urbackup from command line (sudo urbackupsrv run)

Captured the logfile shown below:

sudo urbackupsrv run
2023-05-24 22:07:29: Starting HTTP-Server on port 55414
2023-05-24 22:07:29: HTTP: Server started up successfully!
2023-05-24 22:07:29: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 2 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 31 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 31 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 2 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-05-24 22:07:29: Started UrBackup…
2023-05-24 22:07:29: Removing temporary files…
2023-05-24 22:07:29: Recreating temporary folder…
MOUNT TEST OK
Testing for btrfs…
Create subvolume ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/A’
Create a snapshot of ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/A’ in ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/B’
Delete subvolume (commit): ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/A’
2023-05-24 22:07:29: InternetService: Server started up successfully!
2023-05-24 22:07:29: UrBackup Server start up complete.
2023-05-24 22:07:29: Server started up successfully!
2023-05-24 22:07:29: Looking for old Sessions… 0 sessions
Delete subvolume (commit): ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/B’
BTRFS TEST OK
2023-05-24 22:07:29: Backup destination does handle subvolumes and snapshots. Snapshots enabled for image and file backups.
2023-05-24 22:07:30: Broadcasting on ipv4 interface eth0 addr 192.168.4.50
2023-05-24 22:07:30: Downloading version file…
2023-05-24 22:07:30: Downloading version file…
2023-05-24 22:07:30: Downloading server version info…
2023-05-24 22:07:30: Downloading dataplan database…
2023-05-24 22:08:01: New Backupclient:
2023-05-24 22:08:01: Encrypting with key OTY46j1/23vjo5EGH0G8umkYESNMdzurYlEo1yaKSN9MAwxqAoJ4jroja2scY0lqsXxuZAAAAAAMICHd0Nc83tItcmw3PBQ/ (server)
Bus error

So this is where I’m at. I’ve tried to turn off every option for encryption yet each time it reaches the “Encrypting with key” stage it consistently crashes with “Bus error”

I’m at a complete loss to know what “Bus error” is and thus how to proceed to rectify it. The Raspberry Pi seems to be operating correctly in all other respects. How should I proceed?

Thanks

2

This one perhaps?

I guess you are using the armhf package from the website ? It’ll take forever but maybe you could compile using the debian libcrypto++ ?

https://www.urbackup.org/server_source_install.html

I myself run arm64 debian & urbackup with system libcrypto++ on my Raspberry 4.

3

So you noticed that one way to prevent your original issue is to not store the backups directly into the fs root :slight_smile:

4

Thanks for the quick response.

Your direction has allowed me to refine search terms to find some new information on this topic. I’m not a software engineer, but I can follow a “recipe”. I now understand that this is possibly down to a data misalignment error because of the ARM hardware structure.

You are quite correct. I’m running the armhf version. I shall get the arm64 version and see whether that will work from the binaries directly or whether I will have to compile from source.

I did have a look at compiling UrBackup directly from source code before posting this. ./configure exited (as I expected it to with an unsatisfied dependency). While the title of the missing library is explicit, the challenge, as ever, is to work out what your individual distribution actually calls it.

For clarity though. When you suggest compiling UrBackup against libcrypto++: do you mean compile against the pre compiled package that might be available in the Raspbian repository or compile libcrypto++ from its own source and then compile UrBackup against that?

This leads me to ask the natural question: what might have changed between Buster and Bullseye and/or the UrBackup server versions that have given rise to this error?

5

I have been reading this over and over and over trying to figure out what you meant by this and finally the coin dropped.
I HAVE MINE IN /media/backup witch is a direct mount and I see the problem you point out now (if the drive fail to mount).

Is there an easy way for me to move all my hundreds of backups to /media/backup/urbackups?? Backups are on btrfs correctly making subvols.

Another way is to check if drive is mounted before a backup runs with something like if ! grep -qs '/mnt/backup ' /proc/mounts; then stopurbackupfromrunningsomewhere

Moving the backup storage would be the best.
Please advice.

6

Thanks for the pointers. The server is now back up and running. I installed the arm64 distribution of the Raspberry Pi OS, then tried to install the 64 bit .deb bu that failed on the final setup script because it couldn’t “chown” the backup directory to urbackup:urbackup because of something missing in the script. I ended up compiling from source after satisfying the missing libcurl dependency. Compile from source only took about 30 minutes.

7

I think you might have to do a sudo apt -f install after you have installed the deb the first time to fix if you have missing dependencies, at least that is what I have in my notes since installation.

I use the same procedure (minus the -f install) when updating, did so yesterday without any problems.

wget https://hndl.urbackup.org/Server/2.5.31/urbackup-server_2.5.31_arm64.deb
sudo dpkg -i urbackup-server_2.5.31_arm64.deb
sudo apt -f install #not needed if updating
rm urbackup-server_2.5.31_arm64.deb
8

Hi, I am also having problems since may 24th.
The server was 2.5.27 running on a rpi4 4gb with bulleye version,
the copies are in /media/backup/urbackup.
I have upgraded to 2.5.31 and still have the problem.
in /var/log/urbackup.log since last startup it says the following:

2023-06-07 12:49:27: Starting HTTP-Server on port 55414
2023-06-07 12:49:27: ERROR: HTTP: Failed binding socket to port 55414. Another instance of this application may already be active and bound to this port.
2023-06-07 12:49:27: Started UrBackup…
2023-06-07 12:49:27: Removing temporary files…
2023-06-07 12:49:27: Recreating temporary folder…
2023-06-07 12:49:27: Image mounting disabled: TEST FAILED: guestmount is missing (libguestfs-tools)
2023-06-07 12:49:28: ERROR: InternetService: Failed binding socket to port 55415. Another instance of this application may already be active and bound to this port.
2023-06-07 12:49:28: UrBackup Server start up complete.
2023-06-07 12:49:28: ERROR: Failed binding SOCKET to Port 55413
2023-06-07 12:49:28: ERROR: Error while starting listening to ports. Stopping server.
2023-06-07 12:49:28: Exited Loop
2023-06-07 12:49:28: Deleting at…
2023-06-07 12:49:28: Deleting SelectThreads…
2023-06-07 12:49:28: Deleting lbs…
2023-06-07 12:49:28: Shutting down plugins…
2023-06-07 12:49:28: Deleting server…
2023-06-07 12:49:54: ERROR: Error creating image backup destination.
2023-06-07 12:49:55: ERROR: Error creating image backup destination.
2023-06-07 12:49:55: ERROR: Backing up System Reserved (SYSVOL) partition failed. Image backup failed
2023-06-07 12:49:55: ERROR: Backup failed
2023-06-07 12:49:56: ERROR: Subvolume SRVMARGON/230607-1249 already exists.
2023-06-07 12:49:56: ERROR: Cannot create directory /media/backup/urbackup/SRVMARGON/230607-1249 for backup (server error). File exists (code: 17)
2023-06-07 12:49:56: ERROR: Backup failed
2023-06-07 12:49:56: ERROR: Backing up System Reserved (SYSVOL) partition failed. Image backup failed
2023-06-07 12:49:56: ERROR: Backup failed
2023-06-07 12:49:56: WARNING: Exponential backoff: Waiting at least 40m before next image backup
2023-06-07 12:49:57: WARNING: Exponential backoff: Waiting at least 40m before next file backup
2023-06-07 12:49:57: WARNING: Exponential backoff: Waiting at least 40m before next image backup
2023-06-07 12:52:44: ERROR: No permission to access “/media/backup/urbackup/urbackup_tmp_files”
2023-06-07 12:52:45: ERROR: Error while downloading version info from http://update.urbackup.org/2.5.x/version.txt: Couldn’t connect to server(ec=7),
2023-06-07 12:52:45: ERROR: Error while downloading version info from http://update.urbackup.org/2.5.x/version_linux.txt: Couldn’t connect to server(ec=7),
2023-06-07 12:52:45: ERROR: Error downloading server version information: Couldn’t connect to server(ec=7),
2023-06-07 12:52:45: ERROR: Error downloading dataplan database: Couldn’t connect to server(ec=7),

If I run
sudo urbackupsrv run
from the console I get this:

~/Desktop $ sudo urbackupsrv run
2023-06-07 13:14:20: Starting HTTP-Server on port 55414
2023-06-07 13:14:20: HTTP: Server started up successfully!
2023-06-07 13:14:20: SQLite: recovered 165 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-06-07 13:14:20: SQLite: recovered 827 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2023-06-07 13:14:20: SQLite: recovered 12 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-06-07 13:14:20: SQLite: recovered 19 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-06-07 13:14:20: SQLite: recovered 165 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-06-07 13:14:20: SQLite: recovered 19 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-06-07 13:14:20: SQLite: recovered 827 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2023-06-07 13:14:20: SQLite: recovered 12 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-06-07 13:14:20: Started UrBackup…
2023-06-07 13:14:20: Removing temporary files…
2023-06-07 13:14:20: Recreating temporary folder…
TEST FAILED: guestmount is missing (libguestfs-tools)
2023-06-07 13:14:20: Image mounting disabled: TEST FAILED: guestmount is missing (libguestfs-tools)
Testing for btrfs…
Create subvolume ‘/media/backup/urbackup/testA54hj5luZtlorr494/A’
2023-06-07 13:14:20: InternetService: Server started up successfully!
2023-06-07 13:14:20: UrBackup Server start up complete.
2023-06-07 13:14:20: Server started up successfully!
2023-06-07 13:14:20: Looking for old Sessions… 0 sessions
Create a snapshot of ‘/media/backup/urbackup/testA54hj5luZtlorr494/A’ in ‘/media/backup/urbackup/testA54hj5luZtlorr494/B’
Delete subvolume (commit): ‘/media/backup/urbackup/testA54hj5luZtlorr494/A’
Delete subvolume (commit): ‘/media/backup/urbackup/testA54hj5luZtlorr494/B’
BTRFS TEST OK
2023-06-07 13:14:21: Backup destination does handle subvolumes and snapshots. Snapshots enabled for image and file backups.
2023-06-07 13:14:21: Downloading version file…
2023-06-07 13:14:21: Downloading version file…
2023-06-07 13:14:21: Broadcasting on ipv4 interface eth0 addr 192.168.1.15
2023-06-07 13:14:21: Broadcasting on ipv6 interface eth0 addr fe80::df12:823b:deb8:2c72
2023-06-07 13:14:22: Downloading server version info…
2023-06-07 13:14:22: Downloading dataplan database…
2023-06-07 13:14:53: New Backupclient: SRVMARGON
2023-06-07 13:14:53: New Backupclient: EUGENIO-PC
2023-06-07 13:14:53: New Backupclient: Margon-PC3
2023-06-07 13:14:54: Starting scheduled full image backup of volume “C:”…
2023-06-07 13:14:54: Starting scheduled full image backup of volume “SYSVOL”…
2023-06-07 13:14:54: Starting scheduled full image backup of volume “C:”…
2023-06-07 13:14:54: Starting scheduled full image backup of volume “SYSVOL”…
2023-06-07 13:14:54: Starting scheduled full file backup…
Error del bus

I have also tried to do a clean install on another raspberry pi, and if I install version 2.4.15 it works perfectly.
If I install 2.5.31 it works until I create the clients and when they connect I get the server down.
I have seen that in the Buster version if I have a server on rpi4 with 2.5.30 working fine.

9

Seems like you managed to install 2 versions? Or something else is using that port?

status: sudo systemctl status urbackupsrv.service
Systemd logs: sudo journalctl -u urbacksrv.service

You can also see what is occupying the port with sudo netstat -ltnp | grep -w ':55414'

10

I finally installed the new rasp with version 2.4.15 and it is working without any problem.
I think there is some incompatibility between 2.5.30 and updated bulleye. When you create the clients and they connect for the first time they pull the server down. Even the web page doesn’t load. I will continue with the version that works and I will test with another one that I have to see if they detect what it is and fix it.

11

Glad you got it worked out!

Not sure what you mean by “updated bullseye”, but I am using a raspberry pi 4 with bullseye and urbackup server version 2.5.31, so it def works. I have had that server running for more than a year updating it to latest server software when needed (clients updated from the server). 3 clients connected and doing backups on it, 2 on llinux (ARCH and Debian) and one win11 client.

So there is something funky going on with your setup.
But if what you have now works and you are satisfied with it, go for it!

       _,met$$$$$gg.          bedna@piServer 
    ,g$$$$$$$$$$$$$$$P.       -------------- 
  ,g$$P"     """Y$$.".        OS: Debian GNU/Linux 11 (bullseye) aarch64 
 ,$$P'              `$$$.     Host: Raspberry Pi 4 Model B Rev 1.5 
',$$P       ,ggs.     `$$b:   Kernel: 6.1.21-v8+ 
`d$$'     ,$P"'   .    $$$    Uptime: 4 days, 2 hours, 52 mins 
 $$P      d$'     ,    $$P    Packages: 742 (dpkg) 
 $$:      $$.   -    ,d$$'    Shell: bash 5.1.4 
 $$;      Y$b._   _,d$P'      Terminal: /dev/pts/0 
 Y$$.    `.`"Y$$$$P"'         CPU: BCM2835 (4) @ 1.8GHz 
 `$$b      "-.__              Memory: 469MiB / 7812MiB (6%) 
  `Y$$                        Disk (/): 2.2G / 29G (8%) 
   `Y$$.                      Disk (/media/diverse): 1.7T / 3.6T (48%) 
     `$$b.                    Disk (/media/usr): 7.5G / 24G (34%) 
       `Y$$b.                 Disk (/media/backup): 3.1T / 5.5T (57%) 
          `"Y$b._
              `"""
12

I’m glad to hear that it works correctly with 2.5.31 and bullseye, so I’ll keep trying to configure it to see where I’m going wrong. I was saying about bullseye updated because I had installed version 2.5.27 working correctly, updated with sudo apt-get update, sudo apt-get upgrade, and urbackup stopped working.
I upgraded to 2.5.31 and it still didn’t work.
I reloaded the full 32-bit version of raspbian, installed 2.5.31 and it worked correctly, but when I created the “clients”, downloaded the installer that creates urbackup, installed on windows 10 and when I connected the client machine to the urbackup server on the raspberry, the server stopped working. With clean installation from scratch, and 3 times …
When I follow the same steps, but I put version 2.4.15 I don’t have that problem.
Still, thanks for the heads up so I know I have to keep learning and figure out my problem. :slight_smile:

One more question, can I upgrade a client from 2.4.15 to 2.5.31 and if it doesn’t work go back to 2.4.15 no problem. The only thing I have to do is to save well the configuration of the /var/urbackup/ folder to recover the database configuration. correct?
Thanks

13

Why? And are you installing the 32bit version of the server then?

I think you should probably move on to the 64-bit version of raspbian (or raspberry pi os or whatever they call it now)

Did you install the 32-bit client? I have no idea how a 64-bit client would act with a 32-bit server.

I can’t answer that question for you, sorry.

14

Now I have the problem from yesterday in another Rasp.
I have installed 2.5.28 version.

2023-06-20 13:6:54: ERROR: Error while downloading version info from http://update.urbackup.org/2.5.x/version.txt: Couldn’t connect to server(ec=7)
2023-06-20 13:26:54: ERROR: Error while downloading version info from http://update.urbackup.org/2.5.x/version_linux.txt: Couldn’t connect to server(er(ec=7).
2023-06-20 13:26:54: ERROR: Error downloading server version information: Couldn’t connect to server(ec=7)
2023-06-20 13:26:54: ERROR: Error downloading dataplan database: Couldn’t connect to server(ec=7),

Do you know what could be?

15

Again, WHY???

If you are not going to follow the advice or answer questions there is not really anything anybody can do to help you.

16

I wanted to say that I installed version 2.5.28 when it was the latest version and I had not updated it again. And now suddenly without touching anything I get this error in /var/logs/urbackup.log
And I don’t know why.
Sorry for the error in the translation.
I have 3 raspberry with 3 different urbackup on 3 sites.

17

Ah, I see.

Then that might be why. You try to update over several update versions, I have no idea how that works.

How did you try to update the server? (print the commands you used)

Earlier you were talking about 32-bit rasberry pi os.
Can you clarify what os you have your backup server on (or do you mean you have 3 different backup servers?) What command did you use to install the server/servers?

Can you clarify what os you run on your clients?
Can you clarify what urbackup client software you are using and how you installed them? (again, print commands)

18

Oh, I see.

You were talking about 32 bit rasberry pi os before.
Can you clarify what operating system you have your backup server on (or do you mean you have 3 different backup servers?) What command did you use to install the server/servers?

I go into raspberry pi imager and choose the Raspberry pi OS Full (32 bit) installation.
*I have been setting up 3 raspberry at different stages over time. *
I installed the first one like this, with the above system.
*To install urbackup I downloaded the .deb package from here. *
Install package home:uroni / urbackup-server
I choose version 11 of urbackup-server … deb. that is available.

To install it I run it:
first install btrfs-progs, and format the 5tb hdd with btrfs /dev/sda.
sudo apt-get update
*sudo apt-get install btrfs-progs *
*sudo mkfs.btrfs /dev/sda *
mkdir /media/backup

I add the line to the fstab
/dev/sda /media/backup btrfs defaults,nossd,nofail 0 0
mount the drive
sudo mount /media/backup
I install urbackup like this;
sudo dpkg -i urbackup-server_2.4.15.0-1_armhf.deb (this is the first one I installed)
sudo apt-get -f install

*The first one I installed was 2.4.15 which is still working. The second one also the same version. And the third one already showed version 2.5.27 which was the latest version available. *
Once installed, I set the copy frequency, email alerts, and the only maintenance task I do is to update with sudo apt-get update and upgrade from time to time.

So that could be why. Try upgrading multiple upgrade versions, I have no idea how that works.

How did you try to upgrade the server (print the commands you used)?
to upgrade the server

Can you clarify what operating system you run on your clients?

The operating system can be windows 8/10/11

To create the clients I do it from the urbackup page, add client behind NAT, and put in the computer name. This creates the custom client for me

Can you clarify what urbackup client software you are using and how you installed it (again, print commands)?
to update the server what I do is to log in again to Install package home:uroni / urbackup-server
I download the latest version and install it with dpkg -i urbackup-server…
like when I install.

19

Are your raspberry pies THAT OLD (IIRC since rpi3B they are 64bit cpu:s) or why are you using the 32bit os???

You know, you can download them here UrBackup - Download UrBackup for Windows, GNU/Linux or FreeBSD you don’t need to use the OpenSuse build service, but that should not matter if you choose the correct file.
Change file from arm64 to armhf if you insist on using 32bit.

wget https://hndl.urbackup.org/Server/2.5.31/urbackup-server_2.5.31_arm64.deb
sudo dpkg -i urbackup-server_2.5.31_arm64.deb
sudo apt -f install
rm urbackup-server_2.5.31_arm64.deb

If you insist on keep using the 32 bit os, armhf is the correct one, otherwise, as I really suggest to do, change to the Raspberry Pi OS (64-bit) Lite, or do you actually use the desktop environment for anything? and again, 64 bit not 32.

Do you mean the webpage gui? Or do you mean UrBackup - Download UrBackup for Windows, GNU/Linux or FreeBSD ?

It looks correct, but as I stated before, I would not use 32 but try to use 64 and use the installer from the the link I provided for the clients and then let urbackupserver find the clients automatically over the network (can take a little time, so have patience)

20

Are your raspberry pies THAT OLD (IIRC since rpi3B they are 64bit cpu:s) or why are you using the 32bit os???

*I have been using the 32-bit version because I have always thought it is more stable. Besides, it always gives me access through anydesk to connect for anything and with the 64 bit version it doesn’t work. *

*I have listened to him, I am testing with the 64bit version, with access via teamviewer, and so far it works fine. *
*In windows 10 the client without problem. *
If I have detected on a computer that I have with windows 7 that when installing an older client compatible with windows 7 (version 2.4.11) to make file copies works, but when making image copies I throw down the urbackup server. If I disable the image copies for that pc and leave the file copies it works perfectly.

It looks correct, but as I stated before, I would not use 32 but try to use 64 and use the installer from the the link I provided for the clients and then let urbackupserver find the clients automatically over the network (can take a little time, so have patience)

I was reading the manual again and it is true that it searches the local network every 45 or 50 seconds, so I know more or less how long it takes for the clients to find the server.
I will tell you more news, thanks for your help.
:wink: