Option to unfollow symlinks in Linux backups?

mhohr · November 6, 2014, 2:58pm

Hi there, I haven’t been able to find the option in the settings, but is there simple way or option to unfollow and disable backing up symlink destinations? I’m currently testing urBackup server 1.4.4 and client 1.4.3 on Debian 7.7. I know that I can exclude certain directories, however I would like the choice to not backup/follow symlink destinations.

Any helpful advice would be greatly appreciated.

uroni · November 10, 2014, 8:01pm

Currently not, sorry. At which granularity would you need such a setting (per share or folder?)

mhohr · November 11, 2014, 8:38am

Hi there, thanks for the reply. Ideally I would have liked the simple setting to turn on or off the following/backing up of symlinks. I don’t really need any folder granularity setting to be honest. For the time being, I have found a way around this by adding exclusions to certain folders containing symlinks. This is the only thing that has led me back to using BackupPC for the office. urBackup is a great bit of software and I’ve started using it at home, but due to having multiple servers with shared user spaces in the office, I have many symlinks pointing to some large folders which end up being duplicated when backed up.

chungyan5 · May 12, 2015, 2:57am

I also request this feature too, just need a basic as one flag for all folders with un-follow or follow symlink at this very earlier stage enough.

myii · June 1, 2015, 2:13am

Overview (TL;DR)

This is a long post but I hope it will be beneficial so please bear with me. A quick overview of what I’ve covered below:

Infinite symlink loops: what they look like, example of how Wine symlinks cause them;
Creating a script: description of a script used to investigate and classify symlinks that will be encountered during a UrBackup run;
"Hooking" the script: forcing UrBackup to terminate the backup if a non-excluded, infinite symlink loop is identified;
Script output: links x2 to pastebin.ubuntu.com, showing how it looks (the output of the script, that is);
Script run time: it’s pretty snappy and can be made even snappier;
Linux adoption of UrBackup: why Wine needs special attention if UrBackup is going to be successful working with Linux clients;
Linux backups - include symlinks or not?: the real meat of this issue, bringing some arguments and alternatives (based on BackupPC) that suggest that symlinks should not be followed when backing up Linux clients;
Summary: since this is a ridiculously long posting, I want to make it as easy as possible for you to reply, if you’re still there!

Infinite symlink loops

Had a problem where indexing wouldn’t complete and the file list had reached 300GB after 2 hours!

Uroni said that it looked like infinite symlink loops and to test with tail -f /var/log/urbackup_client.log. Looking into the log, this is just one example of what I found:

ERROR: No permission to stat "/root/.wine/drive_c/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/z:/home/abcde/.byobu/.ssh-agent"

A slightly convoluted path to the .ssh-agent file, then!

Looked at Wine symlinks a bit further (a sample user’s directory):

/home/abcde/.wine/dosdevices:
...
lrwxrwxrwx 1 abcde abcde 10 Jun 22  2013 c: -> ../drive_c
lrwxrwxrwx 1 abcde abcde  8 Jun 22  2013 d:: -> /dev/sr0
lrwxrwxrwx 1 abcde abcde  8 Jul 10  2013 e:: -> /dev/sdb
lrwxrwxrwx 1 abcde abcde 14 Mar 13  2014 f: -> /tmp/tmpfwtyuw
lrwxrwxrwx 1 abcde abcde 10 Mar 13  2014 f:: -> /dev/loop1
lrwxrwxrwx 1 abcde abcde  8 May 12  2014 g:: -> /dev/sda
lrwxrwxrwx 1 abcde abcde  9 Apr 10 01:12 h:: -> /dev/sdb1
lrwxrwxrwx 1 abcde abcde  9 Apr 10 01:12 i:: -> /dev/sdb9
lrwxrwxrwx 1 abcde abcde  1 Jun 22  2013 z: -> /

...

/home/abcde/.wine/drive_c/users/abcde:
...
lrwxrwxrwx  1 abcde abcde 19 Jun 22  2013 Desktop -> /home/abcde/Desktop
lrwxrwxrwx  1 abcde abcde 11 Jun 22  2013 My Documents -> /home/abcde
lrwxrwxrwx  1 abcde abcde 17 Jun 22  2013 My Music -> /home/abcde/Music
lrwxrwxrwx  1 abcde abcde 20 Jun 22  2013 My Pictures -> /home/abcde/Pictures
lrwxrwxrwx  1 abcde abcde 18 Jun 22  2013 My Videos -> /home/abcde/Videos

As soon as I excluded these, the backup ran fine.

Creating a script to identify potential infinite symlink loops

With numerous computers to backup, needed an automated solution. Have managed to put together a shell script (#!/bin/sh) that does the following:

Takes input values for DEFAULT_DIRS and EXCLUDE_FILES in exactly the same format as typed in the UrBackup web interface
Works through each DEFAULT_DIR:
Uses find ${DEFAULT_DIR} -type l to locate all symlinks under that directory; and then with each symlink:
- Captures the symlink target
- Compares the symlink target with each of the DEFAULT_DIRS, to establish if it is:
  1. Inside: target is inside at least one of the DEFAULT_DIRS - harmless
  2. Outside: target is outside all of the DEFAULT_DIRS - potentially troublesome
  3. Infinite symlink loop: target is an ancestor to at least one of the DEFAULT_DIRS - the backup must not proceed unless resolved by an exclusion rule
- The symlink itself is compared to each of the EXCLUDE_FILES to see if it is covered by one of the rules
All of the information is presented using column for formatting - making it easier to see the state of affairs and what needs to be done
Exit message and return code are based on the the number of infinite symlink loops that have not been covered by an exclusion rule

“Hooking” the script into UrBackup to prevent backups where applicable

While useful to be able to manually run to check existing/proposed configurations, really needed this to run before each backup, especially where settings may have been changed before a manual check could be run. Was able to force this in the following way:

Found that the very latest values for DEFAULT_DIRS and EXCLUDE_FILES are available on the client in /usr/local/var/urbackup/data/settings.cfg - improved the script to extract these if called without any arguments
Used the script as /etc/urbackup/prefilebackup so run before indexing/backup
Upon infinite symlink loop error, backup is not allowed to proceed by forcibly killing the UrBackup client and then restarting it after a 5 second delay (this really needs a better solution)
Web interface log shows the error as Constructing of filelist of "{client}" failed - TIMEOUT(1)
Script output is appended to /var/log/urbackup_client.log so that an admin can see what exclusions need to be introduced to get the backup working (since backup wouldn’t work anyway, without the necessary changes)

Enough talk, just show me the output!

Follow the links to see what it looks like.

http://pastebin.ubuntu.com/11484532/:
Conclusion: The backup can proceed.
Here running /etc/urbackup/prefilebackup without arguments, i.e. this would be run via. settings propagated to the client from the UrBackup server
Have ensured that the most important output is right at the bottom - identification of the potential infinite symlink loops and whether they have been resolved via. exclusions
In this case, the exclusion rules */.wine*/dosdevices and */.wine*/drive_c/users/*/My Documents have been sufficient to cover the 3 problems
http://pastebin.ubuntu.com/11484532/:
Conclusion: The backup must be prevented until the infinite symlink loop(s) are resolved.
Here running /etc/urbackup/prefilebackup with arguments for both DEFAULT_DIRS and EXCLUDE_FILES, i.e. running the script manually with test scenarios to see if exclusion rules are sufficient
In this case, have used the same settings as the first run, except have removed */.wine*/dosdevices from the exclusion rules, to see what would happen
Again, looking at the most important output at the bottom, can see that removing the rule has caused 2 of the infinite symlink loops to become “active”
If these were the actual settings on the server, the client would terminate the backup by resetting itself (which to reiterate, might not be a very appropriate way of going about this)

Also note:

The output also displays “outside” symlink targets, the grouping in the middle, which is shown as SYMLINK (MAYBE EXCLUDE: LINK_TARGET outside of backup roots)
These could make it much easier to formulate the exclusion rules, to eradicate issues such as:

How long does the script take to run?

More in seconds, rather than minutes - generally, will be much shorter than the indexing phase of the backup
Considering it saves hours, then probably not a bad trade-off
Running manually from the terminal, can see the progress, to get an idea of how long it would take with different sets of input DEFAULT_DIRS and EXCLUDE_FILES
Many types of filtering available within the script for tweaking; e.g.
ignoring symlinks that point to files makes it much quicker;
can ignore entire directory trees where no concerns are present, such as /etc;
etc., etc. (pun intended)

Linux adoption of UrBackup: Wine symlinks will be problematic

Wine [https://www.winehq.org/] is a very common package installed by many Linux users. By default, it makes multiple symlinks as shown further above. Some of these will definitely cause infinite symlink loops, when backing up user home directories.

I have found that the following exclusion pattern pretty much works for all users’ .wine directories on a system:

*/.wine*/dosdevices;*/.wine*/drive_c/users/*/Desktop;*/.wine*/drive_c/users/*/My Documents;*/.wine*/drive_c/users/*/My Music;*/.wine*/drive_c/users/*/My Pictures;*/.wine*/drive_c/users/*/My Videos;

The additional asterisk in .wine* is to (optimistically) cover all of the wineprefixes [http://wiki.winehq.org/BottleSpec] that users will make - but this is only as good as users sticking to the .wine... directory naming prefix, for all of their wineprefixes!

Ultimately, having to manually add these exceptions is going to lead to more users with problems. Really needs an automated solution of some kind.

Linux backups - include symlinks or not?

With the current system, every change made to the configuration could arbitrarily introduce infinite symlink loops to backups. All it takes is shortening one of the paths in DEFAULT_DIRS or removing the wrong rule from EXCLUDE_FILES. At the very least, “outside” symlinks (see above) could lead to inappropriate directories being backed up - an undesirable outcome.

Having come from BackupPC myself, knowing that I had no such issues with symlinks when using it, I did a little research to understand how it dealt with symlinks.

[http://wiki.zmanda.com/index.php/BackupPC_Client_Configuration]:

$Conf{FtpFollowSymlinks}

This variable controls how BackupPC deals with symlinks. Default is false.

This seemed to be the default when I used it as well - not that I used FTP, though…

[http://backuppc.sourceforge.net/faq/BackupPC.html]:

Linux/Unix

… In fact, by default samba makes symbolic links look like the file or directory that they point to, so you could get an infinite loop if a symbolic link points to the current or parent directory. If you really need to use Samba shares for linux/unix backups you should turn off the “follow symlinks” samba config setting. …)

Similar to our situation; here, turning off “follow symlinks” prevents infinite loops - not that I used Samba, either…

[http://backuppc.sourceforge.net/faq/BackupPC.html]:

Special files

Linux/unix file systems support several special file types: symbolic links, character and block device files, fifos (pipes) and unix-domain sockets. All except unix-domain sockets are supported by BackupPC (there’s no point in backing up or restoring unix-domain sockets since they only have meaning after a process creates them). Symbolic links are stored as a plain file whose contents are the contents of the link (not the file it points to). This file is compressed and pooled like any normal file. Character and block device files are also stored as plain files, whose contents are two integers separated by a comma; the numbers are the major and minor device number. These files are compressed and pooled like any normal file. Fifo files are stored as empty plain files (which are not pooled since they have zero size). In all cases, the original file type is stored in the attrib file so it can be correctly restored.

Emphasis above is mine
This seems to be the clearest explanation of how BackupPC works with symlinks; is there any merit in adopting this method for UrBackup as well?

Me too! But with a bit more besides: as a default setting on Linux, I would prefer not to follow symlinks.

Default: so that I don’t have to remember to turn this off when providing a UrBackup server for multiple Linux clients; please save me from major embarrassment when I have to explain to my client that all of their backups have failed because I forgot to set the little thingummy setting…!
Not: The symlinks’ targets themselves; however, similar to BackupPC, it would be brilliant if each symlinks’ literal contents could be backed up (and be restorable)

Personally, following symlinks is not a feature that is critical - since I wouldn’t be doing my job properly if I didn’t explicitly backup all of the important directories in the first place.

As for granularity, this might be much more complexity for very little gain. If the admin user defines the settings wrongly, he still isn’t immune to all of the potential banana skins. Not to mention the amount of headache involved in trying to work out these settings in the first place (for every single client?!) and then maintain them all - too much micro-management.

Summary: Discussion points / questions

Replying to the above is a waste of time… uh, I mean to say that it will waste your time referencing the issue you’re replying to! So, in a bid to make it as easy as possible for anyone out there who cares to respond, perhaps a numbered list would make things a little easier:

@Uroni @mhohr @chungyan5 @AnyoneElse: Is there any benefit in sharing the script? If so, GitHub? Some other method?
@Uroni: Is there any issue with forcing a client restart to prevent the backup from taking place? Is there a better way of achieving the same goal of preventing the backup?
@Uroni: Following on from #2, I briefly looked through the source code but from my understanding, the prefilebackup is run but not checked for an exit code - with BackupPC, a non-zero exit code from the pre-backup script would prevent the backup from taking place - have I missed something here? Could this be something worth implementing, since it could be useful for a number of situations, if certain conditions are not in place at the time when the backup is going to run…?
@Uroni: Other than not following symlinks, is there a way of automatically recognising and avoiding the multiple symlink-trap that is found in all of the wineprefixes installed on a Linux system?
@Uroni: What do you think about using a symlink policy similar to how BackupPC operates? Symlink content vs. symlink target…
@Everyone: Not following symlinks by default for Linux: is this really, actually a better way?

Linxxs · January 18, 2016, 2:13pm

Hello,
Any chance you could post the scripts you used? They would be helpful to me! Some of my Linux systems actually fill up the drive with “ERROR: Cannot stat “/usr/bin/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/rvim”: Too many levels of symbolic links (40)”

and I believe your script would save me lots of time!

Thank you in advance!

uroni · January 18, 2016, 3:12pm

UrBackup client+server 2.0 has correct symlink handling. You might want to consider joining the beta ( UrBackup Server/Client/Restore 2.0.0 beta )

uroni · January 18, 2016, 3:17pm

And since UrBackup server 1.4.9 you can turn off following symlinks in the advanced settings.

myii · January 18, 2016, 5:21pm

@Linxxs I came here to reply to you but @uroni got there first!

I’ve seen Settings > Advanced > Follow symbolic links on Linux but I haven’t got around to using it yet – good to know that this issue is now resolved, particularly looking forward to using v2.0.

Thanks for all your work @uroni.

Linxxs · January 18, 2016, 5:45pm

thank you, I was so deep in trying to solve the problem, didn’t see that simple fix… I will try it and let you know.
Any chance 2.0 has rear integration?

I am using urbackup for windows and i have to say it works perfectly! Thank you for the software in general!

Stefan · February 15, 2016, 11:34am

Dear Mr uroni,
I have a UrBackup Server v. 2.0.4 beta and a UrBackup Client v. 2.0.4 on a Linux machine (Ubuntu).
I can’t unfollow symlink, on the client logs I have:

Following symbolic link at “/mnt/snapshot/home/stefan/.Private” to “/home/.ecryptfs/stefan/.Private” confirms symlink backup target “.symlink_” to “/”

How can I avoid follow symlink?

Best regards.

uroni · February 15, 2016, 11:53am

I just described it in this post here: UrBackup Server/Client 2.0.4 beta (with Mac OS X client); Restore 2.0.1

HappyCactus · June 30, 2016, 7:46am

Hello,

After having tried bacula, amanda, CrashPlan without success, I tried urbackup and I am almost satisfied.
The only problem I had was on a Linux (Ubuntu) client connecting to a urbackup server on FreeNas.
The indexing was taking forever and was filling out all my partitions - with the indexing file and with the log file.
Finally I found out that the problem was my wine installation and the symlinks loop issue.
Disabling symlinks following that was enabled by default did solve my problem, and surprise! indexing took a few seconds instead of more than 24 hours as the previous time!
As suggested by someone in the previous messages, disabling the “follow symlinks” option by DEFAULT on linux would be very useful for newbies that never faced this problem.
I am using 1.4.9 for client.
Other options are, checking for .wine directories (and other well-known apps) and alert the user for the potential problem. Or checking for loops.
Just my 0.2 cents.

Thank you!
Best Regards

Federico

uroni · June 30, 2016, 10:14am

Or use 2.0.x where this is properly fixed :). Per default it follows symlinks only if the target is not already backed up.

HappyCactus · June 30, 2016, 10:31am

Yes indeed, 2.0.x will be a solution, but if I am not wrong it is in beta status, so not for production, right?
Even if it follows the symlinks only if not already backed up, it will be stuck. Anyway this is not an issue if the user already knows of the problem
Even a timer when indexing takes too long would work. I have a big disk (2 x 1T hd, + 0.5 T SSD) with a LOT of files (compiling yocto and android in many different flavours) but indexing had taken a few seconds when working.

Is there a way to make some directory exclusion on the client side, for example, putting a special named disk (like .no_urbackup) ? it would be very useful…

Thanks

Federico