WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11

Server
1

Database corruption doesn’t seem to be fixed with repair-database.

Server version 2.5.26.

First run:

$ urbackupsrv repair-database
2022-09-27 22:44:26: SQLite: recovered 6537 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 4 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 14 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 6537 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-09-27 22:44:27: Recovering contents of database with id 20...
2022-09-27 22:44:30: SQLite: recovered 14 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-09-27 22:44:30: Recovering contents of database with id 30...
2022-09-27 22:44:30: SQLite: recovered 4 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-09-27 22:44:30: Recovering contents of database with id 23...
2022-09-27 22:44:39: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:48:47: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:50:02: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:54:38: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:54:38: Recovering contents of database with id 24...
2022-09-27 22:54:38: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2022-09-27 22:54:38: Recovering contents of database with id 25...
2022-09-27 22:54:39: Importing database with id 20...
2022-09-27 22:54:45: Importing database with id 30...
2022-09-27 22:54:46: Importing database with id 23...
2022-09-27 23:02:57: Moving rows from lost+found...
2022-09-27 23:02:58: Importing database with id 24...
2022-09-27 23:02:59: Importing database with id 25...

Second run:

$ urbackupsrv repair-database
2022-09-27 23:03:51: Recovering contents of database with id 20...
2022-09-27 23:03:54: Recovering contents of database with id 30...
2022-09-27 23:03:54: Recovering contents of database with id 23...
2022-09-27 23:04:01: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:07:22: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:08:38: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:13:30: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:13:30: Recovering contents of database with id 24...
2022-09-27 23:13:30: Recovering contents of database with id 25...
2022-09-27 23:13:31: Importing database with id 20...
2022-09-27 23:13:42: Importing database with id 30...
2022-09-27 23:13:43: Importing database with id 23...
2022-09-27 23:22:01: Moving rows from lost+found...
2022-09-27 23:22:02: Importing database with id 24...
2022-09-27 23:22:03: Importing database with id 25...

Subsequent runs are the same as the second.

2

defrag-database did not help either…

$ urbackupsrv defrag-database
2022-09-28 05:01:18: Shutting down all database instances...
2022-09-28 05:01:18: Opening urbackup server database...
2022-09-28 05:01:19: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:01:19: Rebuilding Database...
2022-09-28 05:01:38: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:01:38: Rebuilding Database...
2022-09-28 05:01:38: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:01:38: Rebuilding Database...
2022-09-28 05:15:16: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:15:16: Rebuilding Database...
2022-09-28 05:15:16: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:15:16: Rebuilding Database...
2022-09-28 05:15:17: Rebuilding Database successfull.
2022-09-28 05:15:17: Deleting file entry index, if present...
2022-09-28 05:15:17: Done.

Now checking with repair-database:

$ urbackupsrv repair-database
2022-09-28 05:15:41: Recovering contents of database with id 20...
2022-09-28 05:15:45: Recovering contents of database with id 30...
2022-09-28 05:15:45: Recovering contents of database with id 23...
2022-09-28 05:15:51: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:17:47: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:19:05: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:23:42: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:23:43: Recovering contents of database with id 24...
2022-09-28 05:23:43: Recovering contents of database with id 25...
2022-09-28 05:23:44: Importing database with id 20...
2022-09-28 05:23:52: Importing database with id 30...
2022-09-28 05:23:53: Importing database with id 23...
2022-09-28 05:28:37: Moving rows from lost+found...
2022-09-28 05:28:37: Importing database with id 24...
2022-09-28 05:28:39: Importing database with id 25...
3

Same with remove-unknown…

4

You could try to restore the database from the backed up version inside your backupfolder. Save the production db first.

5

Thanks, but the backup is probably corrupted too, since it’s performed daily and the corruption started several days ago.

I’m giving up on this, is there a way to reset the DATA (files) database, but not the METADATA (clients, settings, etc)?

I tried deleting the backup_server_files.db* files and repair/rebuild, but the corruption was still there. What else can I try? I don’t mind losing the data at this point, just not the settings.

6

It’s very intresting, because i have exactly the same error, same line, same code and Debian version 2.5.26
No way to restore backup once again - must have to try fix it…
My backup_server_files is 140 GB and any touch/defrag/dump of database it really problematic

7

2022-11-04 15:34:13: Recovering contents of database with id 30…
2022-11-04 15:34:14: SQLite: recovered 78010 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-11-04 15:34:14: Recovering contents of database with id 23…
2022-11-04 15:34:22: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11

8

urbackupsrv repair-database with 140GB ends with status:

Out of memory: Kill process 20984 (urbackupsrv) score 951 or sacrifice child

9

Oops. How many RAM do you have available? I think, this should not eat much RAM, shouldn’t it? Memory leak?

10

about 16GB of RAM + some swap partiton, definitely not enough for load whole file, but should be enough for normal operation.

anyway, this should not damage the database.

11

We are having the same issue.

@uroni Is it possible that it’s related to the latest update ?

12

Did you managed to fix your problem?

13

Can everyone that thas this problem perhaps post their environment? (Exactly which binary/source version is used, operating system, etc.)
Also perhaps if you use ECC RAM?

The condition hit is

if( pgno>PAGER_MAX_PGNO || pgno==PAGER_MJ_PGNO(pPager) ){

PAGER_MAX_PGNO is 2^31-1, with a default page size of 4KB that would be 8TB, so unlikely that this is this problem.

14

@uroni We are using proxmox as the hypervisor and the VM is running debian!

I think that we are using ECC memory in our servers also!

Debian “buster” using urbackup binary 2.5.27.0

Even the nightly job didn’t finished (stuck at database verification), we need to restart urbackup-server every morning…

15

The solution for that would be to turn off the nightly backup and to back it up differently. If it runs with ECC and on checksummed storage is also doesn’t need to integrity check before backup.

16

yeah but we are still having the “corruption” of the same line. repair and defrag are unable to finish.

17

@uroni I just had a flash about the ECC.

We enable NUMA in our VM a couple of weeks/months ago. Just disable it, reboot the VM and did some test.

The repair-database process is running for the last 8 hours :wink: I still got the WARNING at line 56972 , 7 hours ago and since that, the process is still running and using CPU between 10% and 50% and I can see that the VM is reading the storage at 7MBits/s but have no idea what the process is doing as nothing is written in the /var/urbackup folder.

2022-12-08 07:39:42: SQLite: recovered 8546 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-12-08 07:39:42: SQLite: recovered 47240 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-12-08 07:39:42: SQLite: recovered 6392 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2022-12-08 07:39:42: SQLite: recovered 29276 frames from WAL file /var/urbackup/backup_server_links.db-wal code: 283
2022-12-08 07:39:42: SQLite: recovered 92 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-12-08 07:39:43: SQLite: recovered 8546 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-12-08 07:39:43: Recovering contents of database with id 20...
2022-12-08 07:39:47: SQLite: recovered 92 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-12-08 07:39:47: Recovering contents of database with id 30...
2022-12-08 07:39:47: SQLite: recovered 47240 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-12-08 07:39:47: Recovering contents of database with id 23...
2022-12-08 08:24:30: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11

EDIT: Our VM is running with 6 cores and 32GB RAM!
EDIT2: No “writing” at all on the storage since the WARNING at line 56972

18

Any news on this? Just faced the same issue on a physical machine with Oracle Linux Server 7.9, kernel 5.4.17-2136.309.5.el7uek.x86_64, Intel i3-4170, 16GB.
The backup storage file system is BTRFS, sqlite DBs are kept on a separate volume (LVM on top of RAID1 mdraid)

19

I had this error after every update of urbackup. Doesn’t matter wat Linux version you run or how much memory you have. The only solution for this is to re-install your back-up system with the latest software. After that, don’t update the software anymore otherwise you will face the same issue again.

Someone should investigate and fix this because this is a structural issue and I see this all the time with my clients.

20

What do you mean by that? Remove all backup information and start from scratch?