Servers crashing

fomox · August 4, 2016, 9:27am

Hello,
I got two backup servers getting data from 4 clients. However they seem to crash every day and this is allways the last entries in the log files.

this is last entries from server 1 (10.0.0.30)
2016-08-04 00:25:02: WARNING: HT: No free space available deleting backups…
2016-08-04 00:25:03: ERROR: No permission to access “/media/backup/urbackup//Kontor-HP/160727-1139”
2016-08-04 00:25:03: WARNING: Warning: Directory doesn’t exist: “/media/backup/urbackup//Kontor-HP/160727-1139”

this is the last entries from server 2 (10.0.0.108)
2016-08-03 19:06:28: WARNING: Writer MSMQ Writer (MSMQ) has failure state VSS_WS_FAILED_AT_PREPARE_BACKUP with error VSS_E_WRITERERROR_TIMEOUT. UrBackup will continue with the backu$
2016-08-03 19:06:28: WARNING: Writer MSMQ Writer (MSMQ) has failure state VSS_WS_FAILED_AT_PREPARE_BACKUP with error VSS_E_WRITERERROR_TIMEOUT. UrBackup will continue with the backu$
2016-08-03 19:06:28: WARNING: Writer MSMQ Writer (MSMQ) has failure state VSS_WS_FAILED_AT_PREPARE_BACKUP with error VSS_E_WRITERERROR_TIMEOUT. UrBackup will continue with the backu$
2016-08-03 19:06:28: WARNING: Writer is in error state during snapshot creation. Writer data may not be consistent. This means the files open by this application (e.g. databases) wi$

any ideas?

fomox · August 4, 2016, 11:22am

Server is run on Raspbian on an RPI 3, server version is 2.0.31 and clients are 2.0.30

uroni · August 4, 2016, 11:36am

What is the dmesg line?

This would be really helpfull: https://urbackup.atlassian.net/wiki/display/US/Debugging+with+gdb+on+Linux

fomox · August 4, 2016, 11:46am

where do i find dmesg?

Ill look into the debugging on the link you sendt, but i havent compiled anything myself yet but i`ll try

uroni · August 4, 2016, 11:47am

Enter dmesg (command) as root. There should be something about urbackupsrv.

fomox · August 4, 2016, 11:48am

I cant find anything about urbackup in the output, can i send you the output on an email?

uroni · August 4, 2016, 11:51am

No thanks, maybe you can run the server in the foreground via urbackupsrv run -v debug to see why it crashes?

fomox · August 4, 2016, 11:53am

I`ll try that and post again later, thanks

fomox · August 4, 2016, 11:57am

I get some errors when i try to do this.

2016-08-04 11:56:04: Starting HTTP-Server on port 55414
2016-08-04 11:56:04: HTTP: Server started up successfully!
2016-08-04 11:56:04: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:04: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:04: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:04: ERROR: Error preparing Query [PRAGMA synchronous=NORMAL]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:05: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:05: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:05: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:05: ERROR: Error preparing Query [PRAGMA synchronous=NORMAL]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:06: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:06: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:06: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:06: ERROR: Error preparing Query [PRAGMA synchronous=NORMAL]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:07: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:07: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:07: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:07: ERROR: Error preparing Query [PRAGMA synchronous=NORMAL]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:08: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:08: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:08: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:08: ERROR: Error preparing Query [PRAGMA synchronous=NORMAL]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:09: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:09: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:09: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:09: ERROR: Error preparing Query [PRAGMA synchronous=NORMAL]: unable to open database file
2016-08-04 11:56:09: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:09: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:09: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:09: ERROR: Error preparing Query [PRAGMA cache_size = -2048]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:10: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:10: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:10: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:10: ERROR: Error preparing Query [PRAGMA cache_size = -2048]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:11: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:11: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:11: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:11: ERROR: Error preparing Query [PRAGMA cache_size = -2048]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:12: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:12: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:12: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:12: ERROR: Error preparing Query [PRAGMA cache_size = -2048]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:13: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:13: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:13: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:13: ERROR: Error preparing Query [PRAGMA cache_size = -2048]: unable to open database file. Retrying in 1s…
2016-08-04 11:56:14: WARNING: SQLite: cannot open file at line 31812 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:14: WARNING: SQLite: os_unix.c:31812: (13) open(/var/urbackup/backup_server.db-shm) - errorcode: 14
2016-08-04 11:56:14: WARNING: SQLite: unable to open database file errorcode: 14
2016-08-04 11:56:14: ERROR: Error preparing Query [PRAGMA cache_size = -2048]: unable to open database file
2016-08-04 11:56:14: WARNING: SQLite: cannot open file at line 33278 of [e9bb4cf40f] errorcode: 14
2016-08-04 11:56:14: WARNING: SQLite: os_unix.c:33278: (13) open(/var/urbackup/backup_server_files.db) - errorcode: 14
2016-08-04 11:56:14: Could not open db [urbackup/backup_server_files.db]
2016-08-04 11:56:14: ERROR: Database “urbackup/backup_server_files.db” couldn’t be opened
2016-08-04 11:56:14: ERROR: Couldn’t open backup server database. Exiting. Expecting database at “/var/urbackup/backup_server_files.db”

I tried to run sudo /etc/init.d/urbackupsrv stop first

uroni · August 4, 2016, 12:11pm

Run it as root?

fomox · August 4, 2016, 12:13pm

sorry, now i feel stupid, hehe. thanks again

fomox · August 5, 2016, 7:27am

server 1 (10.0.0.30)
2016-08-04 21:39:05: WARNING: No free space in backup folder. Free space=0 bytes MinFreeSpace=50 MB
2016-08-04 21:39:05: Space to free: 50 MB
2016-08-04 21:39:06: Free space: 0 bytes
2016-08-04 21:39:06: Client with id=5 has 1 full image backups max=2
2016-08-04 21:39:06: Client with id=5 has 0 incremental image backups max=4
2016-08-04 21:39:06: Deleting incomplete file backup ( id=2, backuptime=2016-07-27 11:55:53, path=160727-1139 ) from client “Kontor-HP” ( id=2 ) …
2016-08-04 21:39:06: ERROR: No permission to access “/media/backup/urbackup//Kontor-HP/160727-1139”
2016-08-04 21:39:07: WARNING: Warning: Directory doesn’t exist: “/media/backup/urbackup//Kontor-HP/160727-1139”

server 2 (10.0.0.108)
2016-08-05 03:25:10: Downloading version file…
2016-08-05 03:25:10: Downloading version file…
2016-08-05 03:25:10: Downloading version file…
2016-08-05 03:25:11: Downloading server version info…
2016-08-05 03:25:11: Client with id=4 has 1 full image backups max=5
2016-08-05 03:25:11: Client with id=4 has 0 incremental image backups max=30
2016-08-05 03:25:11: Client with id=3 has 1 full image backups max=5
2016-08-05 03:25:11: Client with id=3 has 0 incremental image backups max=30
2016-08-05 03:25:11: Deleting incomplete file backup ( id=20, backuptime=2016-08-01 21:00:30, path=160801-2049 ) from client “Kontor-HP” ( id=4 ) …
2016-08-05 03:25:11: ERROR: No permission to access “/media/backup/urbackup/Kontor-HP/160801-2049”
2016-08-05 03:25:11: WARNING: Warning: Directory doesn’t exist: “/media/backup/urbackup/Kontor-HP/160801-2049”

I think it crashes when it tries to delete old backups

AckRite · August 5, 2016, 9:26am

Can you verify that urbackup owns those folders on both machines? And for server1, can you check if there’s space on the drive? df -h should help you with that

fomox · August 5, 2016, 9:51am

how do i check that urbackup owns the folders? When i check manually i cant find the folders (i dont think it exists)

server 1 is full but I will remove about 200gb then it should be enough, but the server shouldnt crash even tho it is full?

AckRite · August 5, 2016, 10:27am

When you say crash, do you mean failed backup or?

To check owner, run ls -l. Third and fourth column tells you owner and group, both should be urbackup.
Specifically, check that /media/backup/urbackup belongs to urbackup, then check if Kontor-HP belongs to urbackup too.
Also, one more thing, I notice that Kontor-HP has a slash in front of it - that might be the problem you are after.

fomox · August 5, 2016, 10:41am

crash as in the server crashes (cant access web interface unless i reboot).

everything belongs to root. this is ntfs formatted drives so im not sure access control works?

yes there is an extra slash infront of Kontor-HP, tho i am not sure where it comes from

AckRite · August 5, 2016, 10:47am

Wow, a service crash then, ok. As for ntfs, internet says that it allows access control. In the web interface, what did you set your storage path to? That should belong to urbackup, no question. It doesn’t give you an error there?
About the slash - check client settings for backup paths, maybe there’s an extra slash that should not be there Not sure if that’s important, but my paths don’t end in a slash (/etc;/var and so no and so on).

fomox · August 5, 2016, 10:58am

/media/backup/urbackup no errors on path

K:;G:;C:; these are the paths im backing up