Urbackup server persistently crashes

Server
#1

System

Raspberry Pi 4 4 Gb RAM, 32 Gb SD card running Raspberry Pi OS 11 (Bullseye)

Externally connected array of 4 x 2 Tb hard disks via USB 3 configured as RAID 10

Both Raspberry Pi and disk array are powered via mains electricity filter and UPS

Background

I have been successfully running UrBackup since June 2020, upgrading server and client software as new releases were made available.

A nearby thunderstorm a couple of weeks ago knocked out the power. Power was not restored by the time UPS exhausted. When power returned Raspberry Pi had powered up but disk array remained in standby which caused the next backup to be written to the SD card until it was full.

I tried to recover the mess and in the end gave up and decided to start from scratch.

Next Steps

Installed Raspberry Pi OS 11 on brand new 32 Gb SD card.

Decided to switch the external RAID 10 array from ext4 to btrfs.

Tested that btrfs was working.

Modified fstab to automount external array at required location

Rebooted to check that external array successfully automounted at required location

All fully functional

Made rsync backup of selected directories of Synology NAS using Hyperbackup.

130 Gb backup successful

Installed Urbackup server from repository following instructions on website

Install successful

Noticed that Urbackup kept on crashing (ps ax | grep urb returned no process)

Configured Urbackup as per prompts and modified /etc/default/urbackupsrv log level to “debug”

Restarted Urbackup from command line (sudo urbackupsrv run)

Captured the logfile shown below:

sudo urbackupsrv run
2023-05-24 22:07:29: Starting HTTP-Server on port 55414
2023-05-24 22:07:29: HTTP: Server started up successfully!
2023-05-24 22:07:29: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 2 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 31 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 31 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-05-24 22:07:29: SQLite: recovered 2 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-05-24 22:07:29: Started UrBackup…
2023-05-24 22:07:29: Removing temporary files…
2023-05-24 22:07:29: Recreating temporary folder…
MOUNT TEST OK
Testing for btrfs…
Create subvolume ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/A’
Create a snapshot of ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/A’ in ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/B’
Delete subvolume (commit): ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/A’
2023-05-24 22:07:29: InternetService: Server started up successfully!
2023-05-24 22:07:29: UrBackup Server start up complete.
2023-05-24 22:07:29: Server started up successfully!
2023-05-24 22:07:29: Looking for old Sessions… 0 sessions
Delete subvolume (commit): ‘/mnt/backup/urbackup/testA54hj5luZtlorr494/B’
BTRFS TEST OK
2023-05-24 22:07:29: Backup destination does handle subvolumes and snapshots. Snapshots enabled for image and file backups.
2023-05-24 22:07:30: Broadcasting on ipv4 interface eth0 addr 192.168.4.50
2023-05-24 22:07:30: Downloading version file…
2023-05-24 22:07:30: Downloading version file…
2023-05-24 22:07:30: Downloading server version info…
2023-05-24 22:07:30: Downloading dataplan database…
2023-05-24 22:08:01: New Backupclient:
2023-05-24 22:08:01: Encrypting with key OTY46j1/23vjo5EGH0G8umkYESNMdzurYlEo1yaKSN9MAwxqAoJ4jroja2scY0lqsXxuZAAAAAAMICHd0Nc83tItcmw3PBQ/ (server)
Bus error

So this is where I’m at. I’ve tried to turn off every option for encryption yet each time it reaches the “Encrypting with key” stage it consistently crashes with “Bus error”

I’m at a complete loss to know what “Bus error” is and thus how to proceed to rectify it. The Raspberry Pi seems to be operating correctly in all other respects. How should I proceed?

Thanks

#2

This one perhaps?

I guess you are using the armhf package from the website ? It’ll take forever but maybe you could compile using the debian libcrypto++ ?

https://www.urbackup.org/server_source_install.html

I myself run arm64 debian & urbackup with system libcrypto++ on my Raspberry 4.

#3

So you noticed that one way to prevent your original issue is to not store the backups directly into the fs root :slight_smile:

#4

Thanks for the quick response.

Your direction has allowed me to refine search terms to find some new information on this topic. I’m not a software engineer, but I can follow a “recipe”. I now understand that this is possibly down to a data misalignment error because of the ARM hardware structure.

You are quite correct. I’m running the armhf version. I shall get the arm64 version and see whether that will work from the binaries directly or whether I will have to compile from source.

I did have a look at compiling UrBackup directly from source code before posting this. ./configure exited (as I expected it to with an unsatisfied dependency). While the title of the missing library is explicit, the challenge, as ever, is to work out what your individual distribution actually calls it.

For clarity though. When you suggest compiling UrBackup against libcrypto++: do you mean compile against the pre compiled package that might be available in the Raspbian repository or compile libcrypto++ from its own source and then compile UrBackup against that?

This leads me to ask the natural question: what might have changed between Buster and Bullseye and/or the UrBackup server versions that have given rise to this error?

#5

I have been reading this over and over and over trying to figure out what you meant by this and finally the coin dropped.
I HAVE MINE IN /media/backup witch is a direct mount and I see the problem you point out now (if the drive fail to mount).

Is there an easy way for me to move all my hundreds of backups to /media/backup/urbackups?? Backups are on btrfs correctly making subvols.

Another way is to check if drive is mounted before a backup runs with something like if ! grep -qs '/mnt/backup ' /proc/mounts; then stopurbackupfromrunningsomewhere

Moving the backup storage would be the best.
Please advice.