Unproper cleaning of .directory pool

Manually deleted a few backups from another virtual client then run with remove-unknow.
As i always get this message when i run remove unknown i wasn t paying too much attention. But maybe the link count get unchanged , thus the real data folder is never actually deleted.

It s actually about1500 lines not just the 4 here that are for example
2018-01-15 15:26:43: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/80” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/Bn/BnYQYGu1w51509448359439618489” not found in database. Deleting symlink only.
2018-01-15 15:26:45: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/4c” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/Si/SizJxIPd8x1509448312439570756” not found in database. Deleting symlink only.
2018-01-15 15:26:45: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/ea” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/E8/E8HlDulQuD1509448454439713035” not found in database. Deleting symlink only.
2018-01-15 15:26:45: WARNING: Directory link “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/171030-1710/.hashes/jenkins.dev.sss.com/data/jenkins/workspace/build/resalys/01_create_version/7.8/.svn/pristine/43” with pool path “/var/docker/data/urbackup-server/datas//ct01.dev.sss.com[jenkins]/.directory_pool/P3/P3qblaZjA51509448304439563201” not found in database. Deleting symlink only.

Did you restore a backup of the UrBackup database at some point btw?

By manual deletion you mean deletion from the web interface? (not simply deleting the folder)

Warning can also be caused by a interrupted delete (i.e. power failure) and then be expected behaviour…

Did you restore a backup of the UrBackup database at some point btw?

  • no

By manual deletion you mean deletion from the web interface? (not simply deleting the folder)

  • Had to delete the remaining folder .directory_pool today, all the backups were already deleted, only the client configuration was left, backups were deleted via the ui/

Warning can also be caused by a interrupted delete (i.e. power failure) and then be expected behaviour.

  • It wouldn’t surprise me that the cleanup operation was already interrupted or server shutdown too fast in the past.
    But then I am > 90% sure that the last sequence of : stop, cleanup , start , delete backup , stop , cleanup, give you the logs was done without interuption. Could that be the source of orphan .directory_pool ?

Hello @orogor @uroni .

i’m having the same kind of issue :confused: . recently i change server machine and transferred everything into new server machine.

all clients can see new server and do backup properly. but after changing new server, suddenly clients start taking more space about 25% more compare to previously (i think) as drive fill up.

Some of clients shows only one last backup 10GB (for example) and when i go to .directory pool for that client it shows 50GB (for example). but when i go to web control panel for that client it shows all history and only one last backup (which i can delete from web control panel).

i tried couple of times ruining UNKNOW_REMOVE but doesn’t seem data size reduce.

my question is how can I reduce backup size.?
is it okay if i remove .directory pool folder manually.?

IT’s windows Environment.
server 20.1.20
clients 20.1.17

Many thanks :slight_smile:

HI

If you have a single backup do not delete it … yet
How large do you estimate a single backup should be (on linux ncdu will help, on windows windirstat)?
How large is the full client folder ? (the folder named one level above the .directory pool named as a the client name)

As you have a single backup, both should have similar size, urbackup size may be smaller in case of duplicates files, but not bigger. (well actually if you have alot of small files, because of the .hash, it may be bigger, but you d need to really have a lot of files)
If the urbackup folder is bigger, then i guess you also have some kind of cleaning issue.

Thanks for your reply :slight_smile:

  1. Single full file backup should be around 25GB
  2. the whole client folder is about 250GB

yea looks like there is some kind of cleaning issue, after changing to new machine i found it this issue.

Any suggestion what can i do to resolve this issue ?

many thanks

It is improbable that this affects so many of them. But then I have seen ZFS volumes that have like an hour of (unsynced) updates.

Best would be a series of steps to reproduce the problem. I must admit that I don’t use the directory link method at all currently (instead btrfs).
I’ll do some tests with remove_unknown.

So this is a docker volume, but it has the same path in the docker instance as outside of it? Where do you run urbackupsrv (w/wo remove-unknown)?

Hi

From inside the docker
No, the paths from inside/outside are différents.
These are the paths from inside the urbackup server docker.

We had some issues with btrfs, hence why we arent running it. But i guess we will try again soon.
We try to use btrfs about one time per year, but we always encounter some issue, sometime not from btrfs team fault , but it s mostly btrfs which is affected.
Hence why we use a lot of zfs/ext4, with which we never had problems.

Hi Martin,

Any suggestion would you give me to clear or clean .directory pool.? it takes so much amount of space. i tried exploring all post regarding .directory pool. couldn’t find any answer for it. :frowning:

Many thanks

Hi

Yes this is a “new” issue, as in it has been spotted in this forum post, so investigation is needed.
One of the thing that is needed is the smallest test case than can create this issue. then i guess the fix will be created.

Maybe it happens when backup are deleted manually, or when they are expired, or only with virtual clients,.
Or maybe with more special conditions, like if the server crash at some time in the processing or if the server runs out of space and didn t finish a previous backup.
Or maybe it comes from the folders having specific properties like being hard/softlinked.
Or a combination.

If i have some faith after the office day, i ll try to write a script to produce all theses cases in a vm.
If you have some scripting skills you can also try to so so.

1 Like

Testing shows that remove_unknown does not remove directory links in .directory_pool with no refrences in the database. This can happen if you e.g. use a database backup while having new backups in backup storage. This will be fixed.

Unresolved is why those unreferenced directories exist after normal operation (if they do – obviously they should not). They should be removed during normal file backup removal.

Thanks for the fix.
Maybe that’s because the server was rebooted during a cleanup.

2.2.6 has this cleanup (in remove unknown) now: Server 2.2.6 beta

If you can, please snapshot the backup storage and check if it corrupt any backups (e.g. with urbackupsrv verify-hashes) and revert db and backup storage if it corrupts backups.

remove unknow runned for about 10hours.
it removed a lot of directory pools, maybe 1000-10000

running urbackupsrv verify-hashes
it runs since about 10 hours and is at 15%, it s running io maxed at about 20% iowait, at maybe 50mb/s.
Saved backup should be like 150-200GB
That s maybe not significant as this server is very slow in général.

Verify hash is showing errors like

2018-02-03 21:20:16: ERROR: Error opening file “/data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default/Pepper Data/Shockwave Flash/WritableRoot/#SharedObjects/L98PDDXW/macromedia.com/support/flashplayer/sys/#www.turbo.fr/settings.sol

It s not there :
sudo ls -l “/data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default/Pepper Data/Shockwave Flash/WritableRoot/#SharedObjects/L98PDDXW/macromedia.com”
ls: cannot access ‘/data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default/Pepper Data/Shockwave Flash/WritableRoot/#SharedObjects/L98PDDXW/macromedia.com’: No such file or directory

But the whole backup isn’t missing.
sudo ls -ld “/data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default”
drwxr-x— 23 urbackup urbackup 79 sept. 15 20:20 /data/urbackup2/pascalou[home]/170911-0850/home/orogor/.config/google-chrome/Default

Forgot to get more serious about this and run it in a screen and redirect output. Could these be run from the gui from the advanced tab or so and show a progress bar or just add a progress bar for remove-unknown?

Got a few hundred of them, so apparently it s cleaning too much

2018-02-03 22:50:52: ERROR: Error opening file "/data/urbackup2/pascalou/171009-2223/mame/snap/ws89.png"

ls -l  "/data/urbackup2/pascalou/171009-2223/mame/snap/ws89.png"
ls: cannot access '/data/urbackup2/pascalou/171009-2223/mame/snap/ws89.png': No such file or directory

ls -l  "/data/urbackup2/pascalou/171009-2223/mame/snap"
lrwxrwxrwx 1 urbackup urbackup 75 Oct 21 21:19 /data/urbackup2/pascalou/171009-2223/mame/snap -> /data/urbackup2/pascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256

ls -l  /data/urbackup2/pascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256
ls: cannot access '/data/urbackup2/p    ascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256': No such file or directory

In that case it s not the first or las which is missing

 ls -l /data/urbackup2/pascalou/*/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/170217-1917/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/170429-1704/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/170430-1709/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/170502-2146/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/170503-2151/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/170726-1405/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/170830-1716/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/171104-1723/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/171106-0022/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/171108-2248/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/171111-1246/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/171112-2117/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/171114-1942/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/180201-2203/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/180202-2212/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/180203-0853/mame/snap/ws89.png
-rwxr-x--- 12 urbackup urbackup 9836 Oct 16  2008 /data/urbackup2/pascalou/current/mame/snap/ws89.png

the color is missing , but the set of 3 symlink in the middle is broken
ls -ld /data/urbackup2/pascalou/*/mame/snap
drwxr-x— 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/170217-1917/mame/snap
lrwxrwxrwx 1 urbackup urbackup 75 Apr 16 2017 /data/urbackup2/pascalou/170429-1704/mame/snap -> /data/urbackup2/pascalou/.directory_pool/Sa/SaDGEet7wJ149233821616459840917
lrwxrwxrwx 1 urbackup urbackup 75 Apr 16 2017 /data/urbackup2/pascalou/170430-1709/mame/snap -> /data/urbackup2/pascalou/.directory_pool/Sa/SaDGEet7wJ149233821616459840917
drwxr-x— 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/170502-2146/mame/snap
drwxr-x— 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/170503-2151/mame/snap
drwxr-x— 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/170726-1405/mame/snap
drwxr-x— 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/170830-1716/mame/snap
lrwxrwxrwx 1 urbackup urbackup 75 Oct 21 21:19 /data/urbackup2/pascalou/171009-2223/mame/snap -> /data/urbackup2/pascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256
lrwxrwxrwx 1 urbackup urbackup 75 Oct 21 21:19 /data/urbackup2/pascalou/171012-0957/mame/snap -> /data/urbackup2/pascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256
lrwxrwxrwx 1 urbackup urbackup 75 Oct 21 21:19 /data/urbackup2/pascalou/171016-1426/mame/snap -> /data/urbackup2/pascalou/.directory_pool/Pc/PcnDBKPn0d150861354232735166256
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/171104-1723/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/171106-0022/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/171108-2248/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/171111-1246/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/171112-2117/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/171114-1942/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/180201-2203/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/180202-2212/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/180203-0853/mame/snap
drwx------ 2 urbackup urbackup 3796 Sep 15 2015 /data/urbackup2/pascalou/current/mame/snap

#=================================#
2018-02-03 23:01:30: ERROR: Error opening file “/data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive/vg-system_00045-1167515360.vg”

ls -l "/data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive/vg-system_00045-1167515360.vg"
ls: cannot access '/data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive/vg-system_00045-1167515360.vg': No such file or directory

ls -l "/data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive"
lrwxrwxrwx 1 urbackup urbackup 75 Oct 21 19:08 /data/urbackup2/pascalou/171009-2223/.symlink_/etc/lvm/archive -> /data/urbackup2/pascalou/.directory_pool/Tz/Tzm2NsKUwq150860570232727326683

ls -l  /data/urbackup2/pascalou/.directory_pool/Tz/Tzm2NsKUwq150860570232727326683
ls: cannot access '/data/urbackup2/pascalou/.directory_pool/Tz/Tzm2NsKUwq150860570232727326683': No such file or directory

If you have a snapshot from before remove_unknown could you have a look if it wasn’t already broken before? (for a few of them) You had a post here in the forums a while ago where there were already broken backups.

And yes, with a log of remove_unknown you could have a look at the log with the pool id (something like PcnDBKPn0d150861354232735166256) to see if it was deleted by remove_unknown.

Don’t have the snapshot either. I’ve been way too sloppy on this.

I saved the console output for verify hashes and remove unknown, so it s about 2000 lines for each.
That wasn’t that much interesting. considering i dont have the logs or snapshot to compare.
I looped throught the folder deleted by the the remove unknown that were still in the log s, and couldn’t find an occurrence of them in the verify hashes logs (that may due a lot to not enough logs to do the matching … or not).

However what’s interesting is that i searched for broken symlinks, there s only 2 types of them :
.symlink_void_file and .symlink_void_dir.
Symlinks in the directory pool that point to another directory pool folder.

These kinds of symlinks always match this in the remove-unknown logs : adding a reference, then missing folder

broken symlink :
lrwxrwxrwx 1 urbackup urbackup 81 Oct 13 09:57 ./pascalou[home]/.directory_pool/YN/YNPZ4eEDZj150779477131916395150/bpy1ky4i.default/ImapMail/imap.laposte.net/projects.sbd -> /data/urbackup2/pascalou[home]/.directory_pool/qQ/qQzQTHM4Lf150788142932003053611

remove unknown logs
2018-02-02 19:17:57: WARNING: Adding missing directory reference of pool qQzQTHM4Lf150788142932003053611 to “/data/urbackup2/pascalou[home]/171011-0942/.hashes/home/orogor/.thunderbird/bpy1ky4i.default/ImapMail/imap.laposte.net/projects.sbd”
2018-02-02 19:17:57: ERROR: Cannot open “/data/urbackup2/pascalou[home]/.directory_pool/qQ/qQzQTHM4Lf150788142932003053611”: No such file or directory (2)

So far i canceled the verify hash , the find command for broken link is still running, but i coudn’t match it to something deleted by remove unknown

Ok , canceled the search for broken links, couldn t find one which matched a pool deleted by remove unknown

Ok, so I think the new remove unknown didn’t break anything and the new version probably has a fix such that the corruption should not occur again (with new backups).

Ok, currenty installing a vm, will see if i can corrupt them with test files, if anything happens that’ll be esier to reproduce. :japanese_ogre: