WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11

Database corruption doesn’t seem to be fixed with repair-database.

Server version 2.5.26.

First run:

$ urbackupsrv repair-database
2022-09-27 22:44:26: SQLite: recovered 6537 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 4 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 14 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-09-27 22:44:27: SQLite: recovered 6537 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-09-27 22:44:27: Recovering contents of database with id 20...
2022-09-27 22:44:30: SQLite: recovered 14 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-09-27 22:44:30: Recovering contents of database with id 30...
2022-09-27 22:44:30: SQLite: recovered 4 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-09-27 22:44:30: Recovering contents of database with id 23...
2022-09-27 22:44:39: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:48:47: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:50:02: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:54:38: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 22:54:38: Recovering contents of database with id 24...
2022-09-27 22:54:38: SQLite: recovered 6 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2022-09-27 22:54:38: Recovering contents of database with id 25...
2022-09-27 22:54:39: Importing database with id 20...
2022-09-27 22:54:45: Importing database with id 30...
2022-09-27 22:54:46: Importing database with id 23...
2022-09-27 23:02:57: Moving rows from lost+found...
2022-09-27 23:02:58: Importing database with id 24...
2022-09-27 23:02:59: Importing database with id 25...

Second run:

$ urbackupsrv repair-database
2022-09-27 23:03:51: Recovering contents of database with id 20...
2022-09-27 23:03:54: Recovering contents of database with id 30...
2022-09-27 23:03:54: Recovering contents of database with id 23...
2022-09-27 23:04:01: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:07:22: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:08:38: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:13:30: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-27 23:13:30: Recovering contents of database with id 24...
2022-09-27 23:13:30: Recovering contents of database with id 25...
2022-09-27 23:13:31: Importing database with id 20...
2022-09-27 23:13:42: Importing database with id 30...
2022-09-27 23:13:43: Importing database with id 23...
2022-09-27 23:22:01: Moving rows from lost+found...
2022-09-27 23:22:02: Importing database with id 24...
2022-09-27 23:22:03: Importing database with id 25...

Subsequent runs are the same as the second.

defrag-database did not help either…

$ urbackupsrv defrag-database
2022-09-28 05:01:18: Shutting down all database instances...
2022-09-28 05:01:18: Opening urbackup server database...
2022-09-28 05:01:19: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:01:19: Rebuilding Database...
2022-09-28 05:01:38: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:01:38: Rebuilding Database...
2022-09-28 05:01:38: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:01:38: Rebuilding Database...
2022-09-28 05:15:16: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:15:16: Rebuilding Database...
2022-09-28 05:15:16: Transitioning urbackup server database to different journaling mode...
2022-09-28 05:15:16: Rebuilding Database...
2022-09-28 05:15:17: Rebuilding Database successfull.
2022-09-28 05:15:17: Deleting file entry index, if present...
2022-09-28 05:15:17: Done.

Now checking with repair-database:

$ urbackupsrv repair-database
2022-09-28 05:15:41: Recovering contents of database with id 20...
2022-09-28 05:15:45: Recovering contents of database with id 30...
2022-09-28 05:15:45: Recovering contents of database with id 23...
2022-09-28 05:15:51: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:17:47: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:19:05: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:23:42: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2022-09-28 05:23:43: Recovering contents of database with id 24...
2022-09-28 05:23:43: Recovering contents of database with id 25...
2022-09-28 05:23:44: Importing database with id 20...
2022-09-28 05:23:52: Importing database with id 30...
2022-09-28 05:23:53: Importing database with id 23...
2022-09-28 05:28:37: Moving rows from lost+found...
2022-09-28 05:28:37: Importing database with id 24...
2022-09-28 05:28:39: Importing database with id 25...

Same with remove-unknown

You could try to restore the database from the backed up version inside your backupfolder. Save the production db first.

Thanks, but the backup is probably corrupted too, since it’s performed daily and the corruption started several days ago.

I’m giving up on this, is there a way to reset the DATA (files) database, but not the METADATA (clients, settings, etc)?

I tried deleting the backup_server_files.db* files and repair/rebuild, but the corruption was still there. What else can I try? I don’t mind losing the data at this point, just not the settings.

It’s very intresting, because i have exactly the same error, same line, same code and Debian version 2.5.26
No way to restore backup once again - must have to try fix it…
My backup_server_files is 140 GB and any touch/defrag/dump of database it really problematic

2022-11-04 15:34:13: Recovering contents of database with id 30…
2022-11-04 15:34:14: SQLite: recovered 78010 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-11-04 15:34:14: Recovering contents of database with id 23…
2022-11-04 15:34:22: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11

urbackupsrv repair-database with 140GB ends with status:

Out of memory: Kill process 20984 (urbackupsrv) score 951 or sacrifice child

Oops. How many RAM do you have available? I think, this should not eat much RAM, shouldn’t it? Memory leak?

about 16GB of RAM + some swap partiton, definitely not enough for load whole file, but should be enough for normal operation.

anyway, this should not damage the database.