Manually deleted a few backups from another virtual client then run with remove-unknow.
As i always get this message when i run remove unknown i wasn t paying too much attention. But maybe the link count get unchanged , thus the real data folder is never actually deleted.
It s actually about1500 lines not just the 4 here that are for example
2018-01-15 15:26:43: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/80” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/Bn/BnYQYGu1w51509448359439618489” not found in database. Deleting symlink only.
2018-01-15 15:26:45: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/4c” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/Si/SizJxIPd8x1509448312439570756” not found in database. Deleting symlink only.
2018-01-15 15:26:45: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/ea” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/E8/E8HlDulQuD1509448454439713035” not found in database. Deleting symlink only.
2018-01-15 15:26:45: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/43” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/P3/P3qblaZjA51509448304439563201” not found in database. Deleting symlink only.
Did you restore a backup of the UrBackup database at some point btw?
no
By manual deletion you mean deletion from the web interface? (not simply deleting the folder)
Had to delete the remaining folder .directory_pool today, all the backups were already deleted, only the client configuration was left, backups were deleted via the ui/
Warning can also be caused by a interrupted delete (i.e. power failure) and then be expected behaviour.
It wouldn’t surprise me that the cleanup operation was already interrupted or server shutdown too fast in the past.
But then I am > 90% sure that the last sequence of : stop, cleanup , start , delete backup , stop , cleanup, give you the logs was done without interuption. Could that be the source of orphan .directory_pool ?
i’m having the same kind of issue . recently i change server machine and transferred everything into new server machine.
all clients can see new server and do backup properly. but after changing new server, suddenly clients start taking more space about 25% more compare to previously (i think) as drive fill up.
Some of clients shows only one last backup 10GB (for example) and when i go to .directory pool for that client it shows 50GB (for example). but when i go to web control panel for that client it shows all history and only one last backup (which i can delete from web control panel).
i tried couple of times ruining UNKNOW_REMOVE but doesn’t seem data size reduce.
my question is how can I reduce backup size.?
is it okay if i remove .directory pool folder manually.?
IT’s windows Environment.
server 20.1.20
clients 20.1.17
If you have a single backup do not delete it … yet
How large do you estimate a single backup should be (on linux ncdu will help, on windows windirstat)?
How large is the full client folder ? (the folder named one level above the .directory pool named as a the client name)
As you have a single backup, both should have similar size, urbackup size may be smaller in case of duplicates files, but not bigger. (well actually if you have alot of small files, because of the .hash, it may be bigger, but you d need to really have a lot of files)
If the urbackup folder is bigger, then i guess you also have some kind of cleaning issue.
It is improbable that this affects so many of them. But then I have seen ZFS volumes that have like an hour of (unsynced) updates.
Best would be a series of steps to reproduce the problem. I must admit that I don’t use the directory link method at all currently (instead btrfs).
I’ll do some tests with remove_unknown.
So this is a docker volume, but it has the same path in the docker instance as outside of it? Where do you run urbackupsrv (w/wo remove-unknown)?
From inside the docker
No, the paths from inside/outside are différents.
These are the paths from inside the urbackup server docker.
We had some issues with btrfs, hence why we arent running it. But i guess we will try again soon.
We try to use btrfs about one time per year, but we always encounter some issue, sometime not from btrfs team fault , but it s mostly btrfs which is affected.
Hence why we use a lot of zfs/ext4, with which we never had problems.
Any suggestion would you give me to clear or clean .directory pool.? it takes so much amount of space. i tried exploring all post regarding .directory pool. couldn’t find any answer for it.
Yes this is a “new” issue, as in it has been spotted in this forum post, so investigation is needed.
One of the thing that is needed is the smallest test case than can create this issue. then i guess the fix will be created.
Maybe it happens when backup are deleted manually, or when they are expired, or only with virtual clients,.
Or maybe with more special conditions, like if the server crash at some time in the processing or if the server runs out of space and didn t finish a previous backup.
Or maybe it comes from the folders having specific properties like being hard/softlinked.
Or a combination.
If i have some faith after the office day, i ll try to write a script to produce all theses cases in a vm.
If you have some scripting skills you can also try to so so.
Testing shows that remove_unknown does not remove directory links in .directory_pool with no refrences in the database. This can happen if you e.g. use a database backup while having new backups in backup storage. This will be fixed.
Unresolved is why those unreferenced directories exist after normal operation (if they do – obviously they should not). They should be removed during normal file backup removal.
If you can, please snapshot the backup storage and check if it corrupt any backups (e.g. with urbackupsrv verify-hashes) and revert db and backup storage if it corrupts backups.
remove unknow runned for about 10hours.
it removed a lot of directory pools, maybe 1000-10000
running urbackupsrv verify-hashes
it runs since about 10 hours and is at 15%, it s running io maxed at about 20% iowait, at maybe 50mb/s.
Saved backup should be like 150-200GB
That s maybe not significant as this server is very slow in général.
It s not there :
sudo ls -l “/data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default/Pepper Data/Shockwave Flash/WritableRoot/#SharedObjects/L98PDDXW/macromedia.com”
ls: cannot access ‘/data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default/Pepper Data/Shockwave Flash/WritableRoot/#SharedObjects/L98PDDXW/macromedia.com’: No such file or directory
But the whole backup isn’t missing.
sudo ls -ld “/data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default”
drwxr-x— 23 urbackup urbackup 79 sept. 15 20:20 /data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default
Forgot to get more serious about this and run it in a screen and redirect output. Could these be run from the gui from the advanced tab or so and show a progress bar or just add a progress bar for remove-unknown?
Got a few hundred of them, so apparently it s cleaning too much
2018-02-03 22:50:52: ERROR: Error opening file "/data/urbackup2/pascalou/171009-2223/mame/snap/ws89.png"
ls -l "/data/urbackup2/pascalou/171009-2223/mame/snap/ws89.png"
ls: cannot access '/data/urbackup2/pascalou/171009-2223/mame/snap/ws89.png': No such file or directory
ls -l "/data/urbackup2/pascalou/171009-2223/mame/snap"
lrwxrwxrwx 1 urbackup urbackup 75 Oct 21 21:19 /data/urbackup2/pascalou/171009-2223/mame/snap -> /data/urbackup2/pascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256
ls -l /data/urbackup2/pascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256
ls: cannot access '/data/urbackup2/p ascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256': No such file or directory
In that case it s not the first or las which is missing
ls -l /data/urbackup2/pascalou/*/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/170217-1917/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/170429-1704/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/170430-1709/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/170502-2146/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/170503-2151/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/170726-1405/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/170830-1716/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/171104-1723/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/171106-0022/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/171108-2248/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/171111-1246/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/171112-2117/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/171114-1942/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/180201-2203/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/180202-2212/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/180203-0853/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16 2008 /data/urbackup2/pascalou/current/mame/snap/ws89.png
ls -l "/data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive/vg-system_00045-1167515360.vg"
ls: cannot access '/data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive/vg-system_00045-1167515360.vg': No such file or directory
ls -l "/data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive"
lrwxrwxrwx 1 urbackup urbackup 75 Oct 21 19:08 /data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive -> /data/urbackup2/pascalou/.directory_pool/Tz/Tzm2NsKUwq150860570232727326683
ls -l /data/urbackup2/pascalou/.directory_pool/Tz/Tzm2NsKUwq150860570232727326683
ls: cannot access '/data/urbackup2/pascalou/.directory_pool/Tz/Tzm2NsKUwq150860570232727326683': No such file or directory
If you have a snapshot from before remove_unknown could you have a look if it wasn’t already broken before? (for a few of them) You had a post here in the forums a while ago where there were already broken backups.
And yes, with a log of remove_unknown you could have a look at the log with the pool id (something like PcnDBKPn0d150861354232735166256) to see if it was deleted by remove_unknown.
Don’t have the snapshot either. I’ve been way too sloppy on this.
I saved the console output for verify hashes and remove unknown, so it s about 2000 lines for each.
That wasn’t that much interesting. considering i dont have the logs or snapshot to compare.
I looped throught the folder deleted by the the remove unknown that were still in the log s, and couldn’t find an occurrence of them in the verify hashes logs (that may due a lot to not enough logs to do the matching … or not).
However what’s interesting is that i searched for broken symlinks, there s only 2 types of them :
.symlink_void_file and .symlink_void_dir.
Symlinks in the directory pool that point to another directory pool folder.
These kinds of symlinks always match this in the remove-unknown logs : adding a reference, then missing folder
remove unknown logs
2018-02-02 19:17:57: WARNING: Adding missing directory reference of pool qQzQTHM4Lf150788142932003053611 to “/data/urbackup2/pascalou[home]/171011-0942/.hashes/home/orogor/.thunderbird/bpy1ky4i.default/ImapMail/imap.laposte.net/projects.sbd”
2018-02-02 19:17:57: ERROR: Cannot open “/data/urbackup2/pascalou[home]/.directory_pool/qQ/qQzQTHM4Lf150788142932003053611”: No such file or directory (2)
So far i canceled the verify hash , the find command for broken link is still running, but i coudn’t match it to something deleted by remove unknown
Ok, so I think the new remove unknown didn’t break anything and the new version probably has a fix such that the corruption should not occur again (with new backups).