Maybe use some different kind of disk that doesn’t corrupt the data?
That said, I don’t understand why it would still be corrupt after the repair. That completely rebuilds the database…
Thanks for your reply.
The disks are fine as far as I can tell. They passed the long SMART test and badblocks.
Filesystem is ext4 which is considered safe (will now start from scratch on btrfs).
BTW: Backups and restore seem to be succeeding even with this error. This, in conjunction with the claim that repair completely rebuilds the database, maybe hint that it’s a software error?
Is your system using ECC RAM? Because to me this screams defect RAM.
1 Like
No.
If it is only in backup_server_files.db btw. all it breaks is the file dedup. So it is going to store some files multiple times perhaps – not a big deal.
Would be more worried about hardware issues (disk/RAM) though.
How To Corrupt An SQLite Database File lists the ways this can happen. As for software issues perhaps 5. applies, the best tool to find issues there is to compile with -fsanitize=address.
I think I found the root cause of “WARNING: SQLite: database corruption at line” in my setup. urbackup uses /var/urbackup for sqlite DB files storage. That folder was configured as a symlink to another one. After I replaced it with the bind mount, the issue has disappeared.
I’m using bindfs, you think it matters?
bindfs -u urbackup -g urbackup /storage/var/urbackup /var/urbackup
I’m not aware of bindfs, sorry. I just added this entry into /etc/fstab:
/zmir/urbackup /var/urbackup none defaults,bind 0 0
Yes… now you are adding two new things where there could be bugs, fuse and bindfs. And IMO bugs in fuse and bindfs are more likely than sqlite (e.g. there is an issue that bindfs doesn’t implement fsyncdir which sqlite needs…).
Please just use a bind mount and a standard/popular file system.
It was worth the try, but still…
# urbackupsrv repair-database
2023-10-30 22:23:24: SQLite: recovered 4491 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-10-30 22:23:24: SQLite: recovered 176043 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2023-10-30 22:23:24: SQLite: recovered 28 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-10-30 22:23:24: SQLite: recovered 50 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-10-30 22:23:24: SQLite: recovered 4491 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2023-10-30 22:23:24: Recovering contents of database with id 20...
2023-10-30 22:23:24: SQLite: recovered 50 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2023-10-30 22:23:24: Recovering contents of database with id 30...
2023-10-30 22:23:25: SQLite: recovered 176043 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2023-10-30 22:23:25: Recovering contents of database with id 23...
2023-10-30 22:23:30: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2023-10-30 22:23:31: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2023-10-30 22:23:49: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2023-10-30 22:24:09: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2023-10-30 22:24:09: Recovering contents of database with id 24...
2023-10-30 22:24:09: SQLite: recovered 28 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2023-10-30 22:24:09: Recovering contents of database with id 25...
2023-10-30 22:24:11: Importing database with id 20...
2023-10-30 22:24:13: Importing database with id 30...
2023-10-30 22:24:14: Importing database with id 23...
2023-10-30 22:24:38: Moving rows from lost+found...
2023-10-30 22:24:39: Importing database with id 24...
2023-10-30 22:24:39: Importing database with id 25...
Command used:
mount --bind /storage/var/urbackup /var/urbackup
Filesystem: ext4
Yeah, repair-database still prints the database corruption warning. I’m sorry for misinforming you.
I’ve just tried to follow the instructions from https://urbackup.atlassian.net/wiki/spaces/US/pages/4292612/Troubleshooting+UrBackup+Server+Database and found out that VACUUM command errors with “database or disk is full”. Indeed, sqlite3 uses /var/tmp as a temp folder and it does not have enough free space.
I was able to change the temp folder in sqlite by using PRAGMA temp_store_directory=.
I guess, urbackup might face the same issue. Just set SQLITE_TMPDIR var in /etc/sysconfig/urbackup-server and no more corruption warnings! still see the warning 
It’s disappointing to find out that /var/tmp is being used, since on my setup it is not encrypted. This is why I mounted /var/urbackup to the encrypted storage. Thanks for sharing this.
Me too, and the corruption warnings still show.
In Ubuntu it’s in /etc/default/urbackupsrv, but I can’t confirm the setting works because the newly set directory is empty after starting the service.
I believe you can check the free space (using df) before and during the repair-database command running. If you see the free space is less than before, then that folder is in use
Does anybody found solution for this error? Got it on URBackup running on Windows in Hyper-V.
As I understand it, all administrators of Urbackup instances are eventually confronted with this error: the database is corrupted at line 56972. 
For my part, I discovered this error when another error had just occurred on my current Urbackup instance (hosted on Proxmox Backup Server). But this same error was already present on my previous instance, hosted on Ubuntu Server, and had been for several years if you consider that this error occurs with every update…
However, this error didn’t prevent the application from working perfectly well, so it’s still a consoling piece of information. Just don’t launch the correction tool, and you’ll be left in the dark. 
But what does this phase mean: the database is corrupt at line 56972: which table is affected and in which database?
We don’t know… It’s a shame. Could the error message be improved? 
Thanks in advance to the developers for looking into this bug, which appears to be structural and seems to be present on many instances. 
Ernest.
I’ll also throw my hat into the ring on this as I’m also seeing this error.
# urbackupsrv repair-database -u root
2024-01-10 22:19:55: Recovering contents of database with id 20...
2024-01-10 22:19:55: Recovering contents of database with id 30...
2024-01-10 22:19:55: Recovering contents of database with id 23...
2024-01-10 22:19:57: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2024-01-10 22:20:15: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2024-01-10 22:20:23: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2024-01-10 22:21:39: WARNING: SQLite: database corruption at line 56972 of [3bfa9cc97d] errorcode: 11
2024-01-10 22:21:39: Recovering contents of database with id 24...
2024-01-10 22:21:47: Recovering contents of database with id 25...
2024-01-10 22:21:47: Importing database with id 20...
2024-01-10 22:21:47: Importing database with id 30...
2024-01-10 22:21:47: Importing database with id 23...
2024-01-10 22:23:21: Moving rows from lost+found...
2024-01-10 22:23:21: Importing database with id 24...
2024-01-10 22:23:32: Importing database with id 25...
I’m currently using UrBackup Server 2.5.32 on a Rocky Linux 8.9 LXC container on Proxmox 8.1.3. The datastore for this server is on ZFS and the server has ECC memory, so I have a hard time believing that it’s a corruption issue.
The only reason I ran into this issue is related to troubleshooting a different issue about a client using too much space.
Given the diverse hardware/systems in this thread, and the fact we are all receiving the exact same error, it seems like it shouldn’t be due to corruption, however I’m not exactly sure what that warning message means, so I could be misunderstanding it.
Looking at the code, the function that actually generates the error message in the logs appears to be errorLogCallback, which is registered with sqlite3_config here. It appears that database corruption at line 56972 of [3bfa9cc97d] is coming from SQLite itself.
A cursory Google for “database corruption at line” came up with very little. Perhaps there’s a nugget buried somewhere that can be revealed with better search terms or a direct dive into SQLite’s code.
EDIT: I found (I think) the relevant functions in SQLite’s code. sqlite3CorruptError makes the call to sqlite3ReportError, which is the function that actually prints the output. I’m unable to deduce what lineno is actually referring to, or how I’d go about pulling that data out of the UrBackup SQLite db in order to investigate further.
It is line 56972 of this file https://github.com/uroni/urbackup_backend/blob/2.5.20/sqlite/sqlite3.c
Which is
/* The pager cache has created a new page. Its content needs to
** be initialized. But first some error checks:
**
** (1) The maximum page number is 2^31
** (2) Never try to fetch the locking page
*/
if( pgno>PAGER_MAX_PGNO || pgno==PAGER_MJ_PGNO(pPager) ){
rc = SQLITE_CORRUPT_BKPT;
goto pager_acquire_err;
}
I assume it’s caused by a previous out of memory on temp storage.