Urbackupsrv core dump during repari-database

ffsb · May 11, 2024, 12:36pm

GM,

some of my clients seem unable to complete incremental or full file backups (but images backups are working fine)
Assuming that the issue above was related to a corrupt DB, I have tried to run repair_database but it cause the process to core dump.

Running latest version of urbackupsrv on ubuntu. Clients are running latest version of urbackupclient on windows 11 pro. client with the issue is backing up over Lan.

Here the output of running repair-database:

root@hal:/usr/bin# ./urbackupsrv repair-database  -u urbackup
2024-05-11 08:23:27: SQLite: recovered 8720 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2024-05-11 08:23:27: SQLite: recovered 69408 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2024-05-11 08:23:27: SQLite: recovered 8 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2024-05-11 08:23:27: SQLite: recovered 7 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2024-05-11 08:23:27: SQLite: recovered 8720 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2024-05-11 08:23:27: Recovering contents of database with id 20...
Segmentation fault (core dumped)

Here is the output of dmesg matchingt he core dumps:

[435775.009718] urbackupsrv[2594475]: segfault at 8 ip 000061b0927c8bb1 sp 00007fff3918f550 error 4 in urbackupsrv[61b0926f6000+6de000] likely on CPU 0 (core 0, socket 0)
[435775.009737] Code: c5 02 00 66 0f 1f 44 00 00 48 89 da 48 8d 35 d6 f5 fe ff e8 11 4b 02 00 eb b8 0f 1f 80 00 00 00 00 48 8b 83 c0 11 00 00 89 f5 <4c> 8b 60 08 0f b6 43 0d 84 c0 0f 85 a7 03 00 00 4d 85 e4 74 0b 41
[435963.775823] urbackupsrv[2595645]: segfault at 8 ip 0000623101c98bb1 sp 00007ffff4d51900 error 4 in urbackupsrv[623101bc6000+6de000] likely on CPU 2 (core 2, socket 0)
[435963.775855] Code: c5 02 00 66 0f 1f 44 00 00 48 89 da 48 8d 35 d6 f5 fe ff e8 11 4b 02 00 eb b8 0f 1f 80 00 00 00 00 48 8b 83 c0 11 00 00 89 f5 <4c> 8b 60 08 0f b6 43 0d 84 c0 0f 85 a7 03 00 00 4d 85 e4 74 0b 41
[437267.986114] urbackupsrv[2603462]: segfault at 8 ip 00006141b108bbb1 sp 00007ffcd0ebbe00 error 4 in urbackupsrv[6141b0fb9000+6de000] likely on CPU 1 (core 1, socket 0)
[437267.986141] Code: c5 02 00 66 0f 1f 44 00 00 48 89 da 48 8d 35 d6 f5 fe ff e8 11 4b 02 00 eb b8 0f 1f 80 00 00 00 00 48 8b 83 c0 11 00 00 89 f5 <4c> 8b 60 08 0f b6 43 0d 84 c0 0f 85 a7 03 00 00 4d 85 e4 74 0b 41
[437272.550854] urbackupsrv[2603495]: segfault at 8 ip 000062612bb1abb1 sp 00007ffcbdb000e0 error 4 in urbackupsrv[62612ba48000+6de000] likely on CPU 0 (core 0, socket 0)
[437272.550885] Code: c5 02 00 66 0f 1f 44 00 00 48 89 da 48 8d 35 d6 f5 fe ff e8 11 4b 02 00 eb b8 0f 1f 80 00 00 00 00 48 8b 83 c0 11 00 00 89 f5 <4c> 8b 60 08 0f b6 43 0d 84 c0 0f 85 a7 03 00 00 4d 85 e4 74 0b 41
[515672.058189] urbackupsrv[3069114]: segfault at 8 ip 0000560a5abf7bb1 sp 00007ffc8567a670 error 4 in urbackupsrv[560a5ab25000+6de000] likely on CPU 2 (core 2, socket 0)
[515672.058220] Code: c5 02 00 66 0f 1f 44 00 00 48 89 da 48 8d 35 d6 f5 fe ff e8 11 4b 02 00 eb b8 0f 1f 80 00 00 00 00 48 8b 83 c0 11 00 00 89 f5 <4c> 8b 60 08 0f b6 43 0d 84 c0 0f 85 a7 03 00 00 4d 85 e4 74 0b 41
[515692.933099] traps: InternetService[3031826] general protection fault ip:5759cc933213 sp:73eafefec9c0 error:0 in urbackupsrv[5759cc5a6000+6de000]
[515698.722137] urbackupsrv[3069289]: segfault at 8 ip 000059f80a287bb1 sp 00007ffc65f1a240 error 4 in urbackupsrv[59f80a1b5000+6de000] likely on CPU 0 (core 0, socket 0)
[515698.722159] Code: c5 02 00 66 0f 1f 44 00 00 48 89 da 48 8d 35 d6 f5 fe ff e8 11 4b 02 00 eb b8 0f 1f 80 00 00 00 00 48 8b 83 c0 11 00 00 89 f5 <4c> 8b 60 08 0f b6 43 0d 84 c0 0f 85 a7 03 00 00 4d 85 e4 74 0b 41
[515705.726420] urbackupsrv[3069334]: segfault at 8 ip 00005863e4a0cbb1 sp 00007ffca25f2340 error 4 in urbackupsrv[5863e493a000+6de000] likely on CPU 0 (core 0, socket 0)
[515705.726444] Code: c5 02 00 66 0f 1f 44 00 00 48 89 da 48 8d 35 d6 f5 fe ff e8 11 4b 02 00 eb b8 0f 1f 80 00 00 00 00 48 8b 83 c0 11 00 00 89 f5 <4c> 8b 60 08 0f b6 43 0d 84 c0 0f 85 a7 03 00 00 4d 85 e4 74 0b 41
[

Joe_Knock · May 12, 2024, 2:25pm

Same thing happening in the latest docker container

uroni · May 12, 2024, 3:07pm

Please follow instructions at Recovering Data From A Corrupt SQLite Database for now. But preferably restore the damaged database from backups. Repair might break it even more.

It is best to think of the recovery API as a salvage undertaking. Recovery will extract as much usable data as it can from the wreck of the old database, but some parts may be damaged beyond repair and some rework and testing should be performed prior to returning the recovered database to service.

ffsb · May 12, 2024, 8:07pm

ouch I see…
in that case, is there a simple way to simply wipe out the db completely and let clients start over? in this case I don’t need to get back older files, I am using urbackup in case of hd/ssd failures only.
Ideally preserving the settings though…

Chad_Neeper · June 15, 2024, 1:39pm

Edit: Except for repair-database, my issue probably wasn’t similar to OP’s. About that: It didn’t work on a brand new database either. But, judging by uroni’s answer, he’s already well aware of it.

Just wanted to share my nearly identical experience on a fresh Ubuntu 22.04 LTS with latest updates and latest UrBackup server. My older Ubuntu 20.04 LTS server has failing hardware and so I’m building a new one. Just got UrBackup running yesterday backing up a couple of computers. Just added another handful today and it was humming along without any issues until suddenly for no apparent reason urbackupsrv crashed and refused to restart. Testing showed nearly identical error messages to OP’s.

urbackupsrv repair-database, remove-unknown, and cleanup all fail. I restored the backed up database and discovered that while remove-unknown and cleanup both work, repair-database still failed with the same error messages. So I took it a step further and nuked the database entirely to start fresh. I was surprised to find that repair-database still failed.

Below is the output from my restored database, but the fresh is the same, just fewer frames in the WAL files.

$ sudo urbackupsrv repair-database -u urbackup
2024-06-15 09:12:06: SQLite: recovered 4943 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2024-06-15 09:12:06: SQLite: recovered 16 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2024-06-15 09:12:06: SQLite: recovered 241 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2024-06-15 09:12:06: SQLite: recovered 4943 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2024-06-15 09:12:06: Recovering contents of database with id 20...
Segmentation fault

To be fair, I’m not exactly running on bare metal with this new server, so probably have several potential problem points. I just found it odd that I’m seeing the exact same errors as the OP did around 30 days ago when nobody else that I’ve found seems to have experienced this same error and posted about it. But if someone could at least confirm for me that repair-database should work on a fresh and updated Ubuntu 22.04 LTS with UrBackup installed via the documentation for the Ubuntu install via the PPA, I’d appreciate it!

ffsb · July 4, 2024, 3:23am

FYI, in my case I ended removing everything and restarting backups from scratch (I did reuse the ident keys). so far so good. so I assume it was a on-off glitch db corruption, but it looks like repair-database is not an option.

adamb · July 16, 2024, 9:28am

Hello,
My test configuration: Ubuntu Server 24.04 with Urbackup Server 2.5.33. (Installed from OpenSuSE build service download page)
Only one client with file backups, aprox 60MB of files.
“urbackupsrv repair-database” - the same error: Segmentation fault (core dumped)

Thank you