Remove-unknown can not remove deleted backups

[orogor]

Yestreday I manually initated a delete on half the backups as well as changed retention policy and frequency
Tonight we had an out of space incident, becuase of a mis mounted fs
This morning,i noticed that a lot of the backup that we planned on removing were marked as removed from urbackup UI.
But the files are still present on disk.
The operation should have shrunk the backup from 2.2T to 1.1T, but disk usage didnt changed at all

I dont think it s possible to get more verbosity, in cleanup commands you can’t pass a debug parameter.
So i dont know why it fails.

Please advise, i tried a few things:

As root, with urbackup server stoped :

urbackupsrv remove-unknown
2017-06-15 11:08:30: Going to remove all unknown files and directories in the urbackup storage directory. Waiting 20 seconds…
2017-06-15 11:08:50: Shutting down all database instances…
2017-06-15 11:08:50: Opening urbackup server database…
2017-06-15 11:08:50: SQLite: recovered 1660 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2017-06-15 11:08:50: SQLite: recovered 83017 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2017-06-15 11:08:50: SQLite: recovered 1156 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2017-06-15 11:08:50: SQLite: recovered 34037 frames from WAL file /var/urbackup/backup_server_links.db-wal code: 283
2017-06-15 11:08:50: SQLite: recovered 788 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2017-06-15 11:08:50: SQLite: recovered 1660 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2017-06-15 11:08:50: SQLite: recovered 788 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2017-06-15 11:08:50: SQLite: recovered 83017 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2017-06-15 11:08:50: Testing if backup destination can handle subvolumes and snapshots…
Backupfolder not set
2017-06-15 11:08:50: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.
2017-06-15 11:08:50: Cleaning up 0 percent
2017-06-15 11:08:50: ERROR: Error cleaning up.

urbackupsrv cleanup -a 1500G
2017-06-15 11:09:20: Shutting down all database instances…
2017-06-15 11:09:20: Opening urbackup server database…
2017-06-15 11:09:20: SQLite: recovered 1660 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2017-06-15 11:09:20: SQLite: recovered 83017 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2017-06-15 11:09:20: SQLite: recovered 1156 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2017-06-15 11:09:20: SQLite: recovered 34037 frames from WAL file /var/urbackup/backup_server_links.db-wal code: 283
2017-06-15 11:09:20: SQLite: recovered 788 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2017-06-15 11:09:20: SQLite: recovered 1660 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2017-06-15 11:09:20: SQLite: recovered 788 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2017-06-15 11:09:20: SQLite: recovered 83017 frames from WAL file /var/urbackup/backup_server_files.db-wal code: 283
2017-06-15 11:09:20: Testing if backup destination can handle subvolumes and snapshots…
Backupfolder not set
2017-06-15 11:09:20: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.
#================================
As root with upbackup server started

urbackupsrv remove-unknown
2017-06-15 11:13:49: Going to remove all unknown files and directories in the urbackup storage directory. Waiting 20 seconds…
2017-06-15 11:14:09: Shutting down all database instances…
2017-06-15 11:14:09: Opening urbackup server database…
2017-06-15 11:14:09: Testing if backup destination can handle subvolumes and snapshots…
Backupfolder not set
2017-06-15 11:14:09: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.
2017-06-15 11:14:09: Cleaning up 0 percent
2017-06-15 11:14:09: ERROR: Error cleaning up.

urbackupsrv cleanup -a 1500G
2017-06-15 11:14:11: Shutting down all database instances…
2017-06-15 11:14:11: Opening urbackup server database…
2017-06-15 11:14:11: Testing if backup destination can handle subvolumes and snapshots…
Backupfolder not set
2017-06-15 11:14:11: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.

[ttrammell]

@orogor

2017-06-15 11:09:20: Testing if backup destination can handle subvolumes and snapshots…
Backupfolder not set
2017-06-15 11:09:20: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.

Is your backup path set to the appropriate folder?

[orogor]

Yes, else i wouldn t have 2TB of backup
And i dont know of a way to specify backup path to “urbackupsrv remove-unknown”

[ttrammell]

Do this help?

The article makes mention of either disabling the services manually and running the cleanup, or taking the server offline (off the network) to do the cleanup…

ERROR: Cannot create Directory for backup (Server error)

Hi, uroni. Thank you for the response. I have tried “start_urbackup_server --remove_unknown” as you suggested. The output showed that it removed some files for the “MARLENE-HP” client (the client that is working), but none for “STAFF-HP” (the client that’s not working). In any case, even prior to executing the command, the path “/urbackup/office_backup/STAFF-HP” is already empty. After the clean-up, the situation has not changed much: MARLENE-HP can still perform backups, STAFF-HP still cannot…

[uroni]

What’s the output of running df ?

[orogor]

FS Size Used Avail % mounted on
overlay 47G 8,3G 36G 19% /
tmpfs 3,9G 0 3,9G 0% /dev
/dev/xvda9 47G 8,3G 36G 19% /run
12.12.12.12:/ 8,0E 2,3T 8,0E 1% /var/docker/data/urbackup-server/datas
shm 64M 0 64M 0% /dev/shm
tmpfs 3,9G 0 3,9G 0% /sys/fs/cgroup

[uroni]

@orogor

12.12.12.12:/ 8,0E 2,3T 8,0E 1% /var/docker/data/urbackup-server/datas

Well, that’s just ridiculous. Might be good for marketing purposes, but it breaks something (probably 64bit overflow…). And I suspect even Amazon might not be able to store 8 million terabytes of files.

[orogor]

Considering how costly their storage is , if you start to fill it , they ll just buy more disk and if they run out of mineral on earth , they ll start planetary exploration

[orogor]

Because we keep getting billed for this data, i need to remove it asap.
I know i am being a bit pushy, but can i expect a fix to land in stable in not too long (1-2 weeks).

Or should i script something to compare folders present in UI and delete the absent ones on disks?
Would that break the ref counting, if so can the refcount be fixed at a later time when/if a fix is released?

[uroni]

This might fix it: https://github.com/uroni/urbackup_backend/commit/7e96dfeca55754c205e30ac82b4a4a3413ca2b6d.patch