Repair-database running for a while

Hi,

I think we are having some corruption in our database :

2022-12-06 06:49:19: SQLite: recovered 8287 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-12-06 06:49:22: SQLite: recovered 271367 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-12-06 06:49:23: SQLite: recovered 6387 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2022-12-06 06:49:25: SQLite: recovered 29276 frames from WAL file /var/urbackup/backup_server_links.db-wal code: 283
2022-12-06 06:49:25: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-12-06 06:49:25: SQLite: recovered 8287 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-12-06 06:49:25: Recovering contents of database with id 20...
2022-12-06 06:49:29: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-12-06 06:49:29: Recovering contents of database with id 30...
2022-12-06 06:49:30: SQLite: recovered 271367 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-12-06 06:49:30: Recovering contents of database with id 23...
2022-12-06 07:23:15: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11

We are running the urbackupsrv repair-database and the job is running for the last 3 hours but since the last line @ 7h23, the process is still running and using CPU but nothing more …

The backup_server_files.db is near 425GB, maybe that’s the reason ?

@uroni considering the fact that repair and defrag are having a hard time, my guess is from the backup_server_files.db …

2022-12-06 14:42:06: Shutting down all database instances...
2022-12-06 14:42:06: Opening urbackup server database...
2022-12-06 14:42:08: SQLite: recovered 8287 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-12-06 14:42:30: SQLite: recovered 271367 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-12-06 14:42:32: SQLite: recovered 6387 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2022-12-06 14:42:36: SQLite: recovered 29276 frames from WAL file /var/urbackup/backup_server_links.db-wal code: 283
2022-12-06 14:42:36: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-12-06 14:42:36: SQLite: recovered 8287 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-12-06 14:42:36: Transitioning urbackup server database to different journaling mode...
2022-12-06 14:42:36: SQLite: recovered 8287 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2022-12-06 14:42:36: Rebuilding Database...
2022-12-06 14:42:39: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-12-06 14:42:39: Transitioning urbackup server database to different journaling mode...
2022-12-06 14:42:39: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2022-12-06 14:42:39: Rebuilding Database...
2022-12-06 14:42:40: SQLite: recovered 271367 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-12-06 14:42:40: Transitioning urbackup server database to different journaling mode...
2022-12-06 14:42:44: SQLite: recovered 271367 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2022-12-06 14:42:44: Rebuilding Database...

The last rebuilding is stuck there since over an 1 hour. What’s the best way to start our installation fresh while keeping the config and the clients ?

Roughly how many files do you have for your backup? And what is the total size of the backup?

Also, how many clients are there in total that urbackup server is managing?

While I am new to urbackup, I am still trying to learn more about this as the docs is lacking in terms of explaining all these. I am suspecting there are too many files being managed in backup_server_files.db. Maybe the db is not optimized to handle large files? I am sorry if that is not case.