Follow_symlinks on, but not followed

[DaaNMaGeDDoN]

Hi all, i don’t remember seeing this behavior before, but when i add a backup dir on a client like urbackupclientctl add-backupdir -d ~daanmageddon -n daanmageddon, which holds a symlink ~/daanmageddon/somesymlink The symlink points to a filesystem outside the one where the dir is, like ~daanmageddon/symlink->/mnt/someotherfs/folder. The (incremental file)backup completes successfully, in the backup the symlink file is saved and not the target folder and everything under it.
urbackupclientctl list-backupdirs shows /home/daanmageddon daanmageddon follow_symlinks,symlinks_optional,share_hashes No for the relevant backupdir. I tried using /home/daanmageddon as a path instead of ~daanmageddon and also tried with the setting symlinks_optional removed (as far as i understood this has a different meaning; this makes the backup fail if it isnt able to backup the symlink contents -when follow_symlinks is on- or the symlink files -when follow_symlinks is off). In either case just the symlink file is backed up, but not what is “behind” the symlink. pls note “one_filesystem” is not enabled. The logs do not indicate any issues/warnings with the backup of the symlinks, only some errors related to snapshotting in the DEBUG web view live log, but those i am familiar with (as i have no snapshotting mechanism enabled). The log at /var/log/urbackup.log doesnt even get any updates for such an incremental backup, i guess that one only logs level>=warn. The log at the client side only mentions those snapshot errors, like in the web view live log DEBUG level messages, which are expected (still, maybe technically not because i have disabled snapshotting, but anyway i have seen them before so i suppose that is normal behavior in this scenario - its probably “by design”).

Recently (a whole other story, but goes outside the scope of this thread) i ran into database corruption errors which made me conclude i needed to start over from scratch. So this is quite a “clean” instance. In the sense that i think i maybe have forgotten to enable some setting i had set before, but i cant remember where or what that should be. As far as i recall, only the follow_symlinks setting is what controls this behavior.

To cancel out this is an issue with that particular client, on another client i created a new lv, formatted it, mounted it under /mnt/test, created a file /mnt/test/TESTFILE and linked to it from one already configured backupdir, like /someexistingbackupdir/link-to-test->/mnt/test, of course follow_symlinks is enabled for that (someexsting)backupdir.
In the subsequent backup only the symlink file “link-to-test” is backed up. Via the web interface i can download that file which is just an empty file, while the webview shows its a 18byte file. Investigating the backup store on the backup server i see something very interessting: a symlink that is pointing to ../rootfs/mnt/test ; “rootfs” is the name i gave to another backupdir on the same client, for which i have set one_filesystem, something i have set by default for any client. When i browse into the corresponding folder for that rootfs on the backup store i find that the mount folder for the test volume under /mnt isnt even backed up! So not only is the symlink not folowed but instead rewritten to target another backupdir, which doesnt even seems to have the folder on which the test volume is mounted on, and obviously the data “TESTFILE”, is nowhere in the backupstore. To be clear; i wasnt expecting to find anything under /mnt/test in the backupstore, but the folder “test” isnt even there. I guess this is a combination of trying to be efficient referencing another backupdir, i can see the benefit for that, but it doesnt consider the one_filesystem option for that other backupdir and doesnt backup the folder that is the mountpoint for that filesystem. I just checked and it seems that also on several clients any folders that are used as a mountpoint and have a fs mounted on them are missing. When there is nothing mounted on the folder, it does appear in the backup, weird. I think that was also different before, i remember only seeing the mountpoint, the folder, empty and reminding myself that i needed to include other filesystems as seperate backupdirs. Has that changed?
Maybe i just never noticed this, maybe this is a bug? I remember this behavior was different, i remember instead of the symlink ending up in the backup, the name of the symlink was there but instead of it being a file, it was a folder and in that folder the contents of the target of the symlink. I guess that is intuitive too. Checking another backup of a different client i see indeed in the root of the backups there are no folders like /dev /proc /boot or any other folder that is a mountpoint. I cant expect that to be intentional. Yes this is enabled, but my backupstore is btrfs, also unchecking that makes no difference.

Does anybody else see this and/or has an idea? (PSA: check your backups!)

Running the latest server 2.5.33 on Debian Bookworm bare metal, clients are 2.5.25.

[uroni]

Sounds like a bug.

If a symlinks points into a backup folder that has one_filesystem set and the symlink points at another filesystem than the backup folder it fails to take this into account and does not backup the symlink target. Does that sound correct?

What it needs to do in this case is check if the symlink target is the same filesystem as the backup folder and if not, skip the optimization where it re-uses the existing backup folder.

[DaaNMaGeDDoN]

Hi Martin, thanks for reading my lengthy post! Your description is correct, however the issue is broader. In the sense that the symlink does not have to be on a referenced backupdir for which one_filesystem is enabled, to make things simple i just ran a simple test. I think it speaks for itself:

When i run an incremental backup, this is the result:

As you can see, the lv1 backupdir is backed up, but the symlink is not followed, rather its simply the symlink. Also in the backup the mountpoints for both lv1 and lv2 under /mnt are not backed up). Of course the expected behavior would be that the backup for lv1 would follow the symlink and the “symlink-to-lv” would appear as a folder, which i could just navigate into and see the file “TESTFILE2”.When i download the symlink via the browser:

It is simply an empty file (not even a symlink, minor issue though, as:).

In the backupstore on the urbackup server i see indeed an empty “mnt” folder under the “rootfs” backupdir, confirming mountpoints are not backed up:

and under the backupdir lv1:

A broken link instead of the folder it points to.

As i said, this is not what i remember seeing before. At first i thought it might have something to do with the particular client (possibly mount options, apparmor, etc), but this is for any backup i browse which i happen to know have been relying on following those symlinks. I wonder if this is only something at my end (in which case it could be a setting in my urbackupsrv instance), before we call this a bug i suppose we need to cancel that out.

So in short: symlinks dont appear to be followed when follow_symlinks is set and mountpoint(folders) are not backed up as empty folders when the one_filesystem attribute is set, i believe both are not intentional or desirable, possibly a bug, possibly something at my end. I’d love to hear if others see this (new?) behavior too, if not i should of course investigate further at my end. If it is, than that is scary stuff for anybody out there that has been under the impression their backups follow symlinks. I hope its just something at my end!

The other behavior, where the symlink points to a folder that is coincidentally part of another backupdir that has one_filesystem enabled and the symlink is rewritten (i suspect intentionally) to link to that other backupdir instead, and the subsequent backup does not hold the data that is behind that symlink, is possibly the same issue and just a narrowed down version of it, possibly a more expected “bug”, since it referred to a backupdir that has the one_filesystem attr set. In any case i dont think that is intentional nor desired too.

Let’s find out im the only one with this issue first.

[DaaNMaGeDDoN]

I have noticed one thing that i needed to add; i have one client that uses btrfs snapshotting, with that client i do see the (empty) mountpoints being backed up, like folders under /mnt, but also /proc /sys /dev etc. This is different from the rest of the clients that do not use snapshotting. And if i may be free to add another remark, the client that does use snapshotting reports failures in the logs, for the /boot fs which is ext4, although its contents are backed up fine and the mountpoint does appear in the backup for the rootfs. The error is right; ext4 (without lvm) isnt “snapshottable”, but the error in the log is misleading; intuitively i would think urbackup would only try to use snapshotting for backupdirs that use a fs that allows to make snapshots. So when you enable snapshotting it will throw errors for any that is not, even though it sucesfully falls back to regular “file lookup backups”. For clarity the backupdirs configured on that client are the same as in the screenshot i provided in my previous update. I will need to test how it handles reflinks on a “snapshottable” backupdir, to see if behvior is also different for that on “snapshottable” backupdirs that have reflinks and follow_reflink is on.

EDIT:
i ran a similar test to the one i ran before, but now on the client that uses snapshots. I created 2 lvs, formatted them both btrfs, mounted under /mnt/lv1 and /mnt/lv2. Created a symlink /mnt/lv1/symlink-to-lv2->/mnt/lv2 and a 20MB file /mnt/lv2/TESTDATA. Added the backupdir for /mnt/lv1 with follow_symlinks set. The symlink is not followed, the TESTDATA is not in the backup. The symlink is however a folder in the backup (with no contents though), like so:

In conclusion, the behavior for mountpoints is different for backupdirs where snapshotting is enabled/possible, but symlinks are not followed too.
Could it be because the mountpoint or symlink is always under the rootfs backupdir, that symlinks are not followed? I guess i will need to run another test to cancel that out.

[DaaNMaGeDDoN]

(Created a new response to help visibility)
I ran another test with very interesting results!
I temporarily disabled the rootfs and bootfs backupdir like so:

For clarity on the client (at that moment):

Then i ran another incremental update, and now:

It explicitly says it it following the symlink.
BUT, when i check the backup via the webinterface i see this:

Again, the symlink is backed up as a symlink, not followed, however when i go on the urbackup server and browse the backupstore and go into the folder for that last backup, i DO see the a “.symlink_lv2” under that, if i go into that i see the file TESTDATA:

Maybe indeed this is a bug, possibly with my findings you will realize immediately what is going on Martin? Feels indeed the rewriting of the symlinks is still referring to a backupdir that doesnt follow symlinks and as such fails, in this last case it even refers to a backupfolder that isnt even part of the backup.