Awfully slow file backups, SQLite protocol errorcode 15

Hello.

One of my backup servers, which was working fine for months, is now malfunctioning quite badly. As a consequence clients are reporting activities ETA in weeks and are unable to finish backups.
Speed reported on Activities tab is awful and flat at 0 most of the time. These partial jobs stick on Activities tab for days even if client went offline.

Comparing logfiles between my UrBackup servers one thing sticks out - a lot of SQLite protocol errorcodes (15) on this problematic machine, for example:

2020-03-18 03:53:18: ERROR: SQLITE: Long running query Stmt: [INSERT INTO files (backupid, fullpath, hashpath, shahash, filesize, rsize, client
id, incremental, next_entry, prev_entry, pointed_to) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
2020-03-18 15:56:29: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 15:56:30: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 15:56:31: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 15:56:32: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 15:56:37: WARNING: SQLite: statement aborts at 18: [SELECT b.id AS backupid, m.id AS id, path, m.mounttime AS mounttime, partition FROM (mounted_backup_images m INNER JOIN backup_images b ON m.backupid=b.id) WHERE m.mounttime!=0 AND m. errorcode: 15
2020-03-18 16:07:21: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 16:07:21: WARNING: SQLite: statement aborts at 18: [SELECT b.id AS backupid, m.id AS id, path, m.mounttime AS mounttime, partition FROM (mounted_backup_images m INNER JOIN backup_images b ON m.backupid=b.id) WHERE m.mounttime!=0 AND m. errorcode: 15
2020-03-18 16:07:21: WARNING: SQLite: statement aborts at 33: [SELECT c.id AS clientid, c.name AS clientname, file_ok, image_ok, alerts_state, strftime(‘%s’, lastbackup) AS lastbackup, strftime(‘%s’, lastseen) AS lastseen, strftime(‘%s’, lastbacku errorcode: 15
2020-03-18 16:07:22: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 16:07:23: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 16:07:24: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 16:07:25: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 16:07:26: WARNING: SQLite: statement aborts at 27: [UPDATE backups SET running=CURRENT_TIMESTAMP WHERE id=?] locking protocol errorcode: 15
2020-03-18 16:08:21: WARNING: SQLite: statement aborts at 16: [SELECT id, send_to, subject, message, next_try, retry_count FROM mail_queue WHERE next_try IS NULL or next_try>=?] locking protocol errorcode: 15

Unfortunately I don’t have a clue what it means.

I’ve tried to set up everything from scratch - server was reinstalled with a different OS (from FreeBSD to Ubuntu Bionic) and UrBackup clients were also reinstalled. But the problem persists.

S.M.A.R.T. tests are passing fine.

I’m using ZFS pool for data storage, this server has about 20 Windows clients. UrBackup Server and client at latest versions from ppa repository.

Any ideas, please?

Could you run sqlite_analyzer from sqlite-tools at https://www.sqlite.org/download.html with backup_server.db and post the output? Thanks!

@uroni it looks like the output is too long for this forum so I pasted it here https://pastebin.com/ZACR8C1d
Let me know if you prefer it somewhere else or in a different format.

Turns out it was a tricky hardware issue - faulty hard drive controller or cabling.

I’m seeing some progress now after HW replacement so I’ll close this thread for now.