Problem with internal database

general discussion
1

Hi,

I’ve encountered a problem with internal database of UrBackup server. On status page, urbackup server shows:

On disk where databases are stored, there is enough free space.

I’ve issued command:

echo ".dump" | sqlite3 /var/urbackup/backup_server.db

The last line of generated SQL script shows:

ROLLBACK; -- due to errors

I’ve tried to recover backup_server.db file from the backup (by overwriting /var/urbackup/backup_server.db with file stored in backup directory), but after first client is backed up (full file backup), An error occured while accessing or checking UrBackup’s internal database. show again.

What can be a reason of this problem and how can I resolve it?

The UrBackup server runs on virtual machine with 1GB RAM, Ubuntu 12.04 powered. Backups are stored on NFS mount. UrBackup databases are stored in /var/urbackup which resides on virtual drive. UrBackup server version 1.3.2.466-1ubuntu1~precise

2

Can you have a look at the log file, to see when it realizes it is corrupted?

Can you check if you can dump the backup?

3

First logs from /var/log/urbackup indicating problems with database:

2014-04-14 13:31:18: ERROR: SQL: database disk image is malformed Stmt: [SELECT id, (MAX(b.bytes_used_files)+MAX(b.bytes_used_images)) AS used, strftime('%d.%m', MAX(b.created), 'localtime') AS tdate, strftime('%d', MAX(b.created), 'localtime') AS tdate_short FROM (SELECT MAX(created) AS mcreated FROM (SELECT * FROM clients_hist WHERE  0=0 AND created>date('now','-1 month') ) GROUP BY strftime('%d.%m', created, 'localtime'), id HAVING mcreated>date('now','-1 month') ORDER BY mcreated DESC LIMIT 600 ) a INNER JOIN clients_hist b ON a.mcreated=b.created WHERE  0=0 AND b.created>date('now','-1 month') GROUP BY strftime('%d.%m', b.created, 'localtime'), id ORDER BY b.created DESC]  

(...)  

2014-04-14 13:35:32: ERROR: Error in CQuery::Execute - database disk image is malformed  Stmt: [INSERT INTO files_new (backupid, fullpath, hashpath, shahash, filesize, created, rsize, clientid, incremental) SELECT backupid, fullpath, hashpath, shahash, filesize, created, rsize, clientid, incremental FROM files_tmp]

What do you mean by dumping the backup?

I’ve dumped databases in /var/log/urbackup:

echo ".dump" | sqlite3 ./backup_server_settings.db > /tmp/backup_server_settings.sql
echo ".dump" | sqlite3 ./backup_server.db > /tmp/backup_server.sql
echo ".dump" | sqlite3 ./cache/backup_server_files_cache.db > /tmp/backup_server_files_cache.sql

Only /tmp/backup_server.sql ends with ‘ROLLBACK; – due to errors’.

Also, in logs, I’ve seen problems with creating hard links to files.

4

In the mean time I’ve reinstalled urbackup_server:

sudo apt-get purge urbackup-server
sudo apt-get install urbackup-server

I’ve stopped the server:

sudo /etc/init.d/urbackup_srv stop

I’ve also copied old ‘server_ident.key’ and ‘server_token.key’ to ‘/var/urbackup’ and cleaned backup directory (where all data are stored). I didn’t reinstall clients.

Next, I’ve started urbackup server:

sudo /etc/init.d/urbackup_srv start

As expected, all connected clients started to perform full file backup. After some time, I’ve checked server status and it again informed me that it has some problem with internal database. This time there were no problems with internal database logged into ‘urbackup.log’ (full log pasted at the end of this post). The log indicates database upgrades (as WARNING) and some errors. One of them is:

2014-04-15 11:20:11: ERROR: No permission to access "/media/backup/urbackup/urbackup_tmp_files"

But I’ve checked and ‘urbackup’ user has write access to this directory.

Also there were no ‘ROLLBACKs’ when I’ve dumped databases. I’ve switched to activities tab in WEB interface and server didn’t respond - it looks like it hangs.

Contents of ‘/var/log/urbackup.log’:

2014-04-15 11:20:07: WARNING: Upgrading...
2014-04-15 11:20:07: WARNING: Converting database to journal mode...
2014-04-15 11:20:07: WARNING: Upgrading database to version 8
2014-04-15 11:20:07: WARNING: Upgrading database to version 9
2014-04-15 11:20:07: WARNING: Upgrading database to version 10
2014-04-15 11:20:08: WARNING: Upgrading database to version 11
2014-04-15 11:20:08: WARNING: Upgrading database to version 12
2014-04-15 11:20:08: WARNING: Upgrading database to version 13
2014-04-15 11:20:08: WARNING: Upgrading database to version 14
2014-04-15 11:20:08: WARNING: Upgrading database to version 15
2014-04-15 11:20:09: WARNING: Upgrading database to version 16
2014-04-15 11:20:09: WARNING: Upgrading database to version 17
2014-04-15 11:20:09: WARNING: Upgrading database to version 18
2014-04-15 11:20:10: WARNING: Upgrading database to version 19
2014-04-15 11:20:10: WARNING: Upgrading database to version 20
2014-04-15 11:20:10: WARNING: Upgrading database to version 21
2014-04-15 11:20:10: WARNING: Upgrading database to version 22
2014-04-15 11:20:10: WARNING: Upgrading database to version 23
2014-04-15 11:20:10: WARNING: Upgrading database to version 24
2014-04-15 11:20:10: WARNING: Upgrading database to version 25
2014-04-15 11:20:10: WARNING: Upgrading database to version 26
2014-04-15 11:20:10: WARNING: Upgrading database to version 27
2014-04-15 11:20:10: WARNING: Upgrading database to version 28
2014-04-15 11:20:10: WARNING: Done.
2014-04-15 11:20:11: ERROR: No permission to access "/media/backup/urbackup/urbackup_tmp_files"
2014-04-15 11:20:26: WARNING: Shutting down (Signal 15)
2014-04-15 11:26:45: ERROR: Sending settings to client failed
2014-04-15 11:26:46: ERROR: Sending settings to client failed
2014-04-15 11:26:47: ERROR: Sending settings to client failed
2014-04-15 11:26:48: ERROR: Sending settings to client failed
2014-04-15 11:26:49: ERROR: Sending settings to client failed
2014-04-15 11:26:50: ERROR: Sending settings to client failed
2014-04-15 11:26:51: ERROR: Sending settings to client failed
2014-04-15 11:26:52: ERROR: Sending settings to client failed
2014-04-15 11:26:53: ERROR: Sending settings to client failed
2014-04-15 11:27:18: WARNING: Removing reference because restart own was specified and only own tokens are present
2014-04-15 11:27:21: WARNING: Removing reference because restart own was specified and only own tokens are present
2014-04-15 11:27:21: ERROR: Writer MSDEWriter has failure state VSS_WS_STABLE with error VSS_E_WRITERERROR_NONRETRYABLE. UrBackup will continue with the backup but the associated data may not be consistent.
2014-04-15 11:27:25: ERROR: Writer MSDEWriter has failure state VSS_WS_STABLE with error VSS_E_WRITERERROR_NONRETRYABLE. UrBackup will continue with the backup but the associated data may not be consistent.
2014-04-15 11:27:25: WARNING: Removing reference because restart own was specified and only own tokens are present
2014-04-15 11:27:25: WARNING: Removing reference because restart own was specified and only own tokens are present
2014-04-15 11:31:09: ERROR: Error getting file "lHmyEFgC3iZ2oyMlSxVv|Projekty - Piotr/Gwizdek/Ekshumacja.odt" from piotr-Komputer. Errorcode: FILE_DOESNT_EXIST (3)
2014-04-15 11:31:09: ERROR: Error getting file "lHmyEFgC3iZ2oyMlSxVv|Projekty - Piotr/Gwizdek/Ekshumacja.pdf" from piotr-Komputer. Errorcode: FILE_DOESNT_EXIST (3)
2014-04-15 11:31:46: ERROR: Error getting file "lHmyEFgC3iZ2oyMlSxVv|Projekty - Piotr/Gwizdek/próba.odt" from piotr-Komputer. Errorcode: FILE_DOESNT_EXIST (3)
2014-04-15 11:50:42: ERROR: HT: Error getting free space for path "/media/backup/urbackup/tomek/140415-1126/praca/KriostatDoroty/mango4kd/apache-tomcat-6*0.20/webapps/ROOT/WEB-INF/dox/pt/eventDetectors.htm"
2014-04-15 12:01:32: ERROR: Connecting to ClientService of "SC2" failed: Sending settings to client failed
2014-04-15 12:14:35: WARNING: HT: Error creating hardlink from "/media/backup/urbackup/tomek/140415-1126/praca/eclipse_workspace/Mango/war/reskurces/dojo/demos/widget/Mail/MailAccount.html" to "/media/backup/urbackup/tomek/140415-1126/praca/eclipse_workspace/ScadaBR/WebContent/resources/dojo/demos/widget/Mail/MailAccount.html"
5

FYI, I started to suspect that the reason of the problems is virtual machine I use. Now I’m testing this case.

6

It looks like changing VirtualBox version (downgrading) solved the problem. To sum up, I use package urbackup-server (Version: 1.3.2.466-1ubuntu1~precise) on Ubuntu 12.04 LTS installed on virtual machine:
[list]
[:3rn5raed] There are some strange problems with UrBackup Server if guest OS runs on virtualbox-4.3 (Version: 4.3.10-93012~Ubuntu~precise).[/:m:3rn5raed]
[:3rn5raed] There are NO known problems if guest OS runs on virtualbox-4.2 (Version: 4.2.24-92790~Ubuntu~precise).[/:m:3rn5raed][/list:u:3rn5raed]
Host OS is Ubuntu 12.04 LTS, for both virtual machines I’ve used the same VMDK image.

7

Well good thing it is properly handling/detecting corruption now.

VirtualBox is unstable in my experience as well (I still use it). The Linux Kernel developers don’t even accept bug reports anymore if the virtual box guest driver module is loaded.

8

How can i open or dump an encrypted databases:

My hardware is A Thecus NAS N4200Pro:

my sql-software is:
sqlite -version
2.8.15

Pfad der Urbackup-Dateien: /raid/data/urbackup/urbackup/

Befehl

Blockquote echo “.dump” | sqlite /raid/data/urbackup/urbackup/backup_server.db > /tmp/urb.db

ergibt:

Blockquote Unable to open database “/raid/data/urbackup/urbackup/backup_server.db”: file is encrypted or is not a database

9

Try a more recent sqlite version. It is a sqlite3 database with wal and other features.
From the backup location you can just copy the database (it is not open continuously).