Hi there, I haven’t been able to find the option in the settings, but is there simple way or option to unfollow and disable backing up symlink destinations? I’m currently testing urBackup server 1.4.4 and client 1.4.3 on Debian 7.7. I know that I can exclude certain directories, however I would like the choice to not backup/follow symlink destinations.
Any helpful advice would be greatly appreciated.
Currently not, sorry. At which granularity would you need such a setting (per share or folder?)
Hi there, thanks for the reply. Ideally I would have liked the simple setting to turn on or off the following/backing up of symlinks. I don’t really need any folder granularity setting to be honest. For the time being, I have found a way around this by adding exclusions to certain folders containing symlinks. This is the only thing that has led me back to using BackupPC for the office. urBackup is a great bit of software and I’ve started using it at home, but due to having multiple servers with shared user spaces in the office, I have many symlinks pointing to some large folders which end up being duplicated when backed up.
I also request this feature too, just need a basic as one flag for all folders with un-follow or follow symlink at this very earlier stage enough.
This is a long post but I hope it will be beneficial so please bear with me. A quick overview of what I’ve covered below:
Had a problem where indexing wouldn’t complete and the file list had reached 300GB after 2 hours!
Uroni said that it looked like infinite symlink loops and to test with tail -f /var/log/urbackup_client.log. Looking into the log, this is just one example of what I found:
ERROR: No permission to stat "/root/.wine/drive_c/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/c:/users/root/My Documents/.wine/dosdevices/z:/home/abcde/.byobu/.ssh-agent"
.ssh-agent file, then!Looked at Wine symlinks a bit further (a sample user’s directory):
/home/abcde/.wine/dosdevices:
...
lrwxrwxrwx 1 abcde abcde 10 Jun 22 2013 c: -> ../drive_c
lrwxrwxrwx 1 abcde abcde 8 Jun 22 2013 d:: -> /dev/sr0
lrwxrwxrwx 1 abcde abcde 8 Jul 10 2013 e:: -> /dev/sdb
lrwxrwxrwx 1 abcde abcde 14 Mar 13 2014 f: -> /tmp/tmpfwtyuw
lrwxrwxrwx 1 abcde abcde 10 Mar 13 2014 f:: -> /dev/loop1
lrwxrwxrwx 1 abcde abcde 8 May 12 2014 g:: -> /dev/sda
lrwxrwxrwx 1 abcde abcde 9 Apr 10 01:12 h:: -> /dev/sdb1
lrwxrwxrwx 1 abcde abcde 9 Apr 10 01:12 i:: -> /dev/sdb9
lrwxrwxrwx 1 abcde abcde 1 Jun 22 2013 z: -> /
...
/home/abcde/.wine/drive_c/users/abcde:
...
lrwxrwxrwx 1 abcde abcde 19 Jun 22 2013 Desktop -> /home/abcde/Desktop
lrwxrwxrwx 1 abcde abcde 11 Jun 22 2013 My Documents -> /home/abcde
lrwxrwxrwx 1 abcde abcde 17 Jun 22 2013 My Music -> /home/abcde/Music
lrwxrwxrwx 1 abcde abcde 20 Jun 22 2013 My Pictures -> /home/abcde/Pictures
lrwxrwxrwx 1 abcde abcde 18 Jun 22 2013 My Videos -> /home/abcde/Videos
With numerous computers to backup, needed an automated solution. Have managed to put together a shell script (#!/bin/sh) that does the following:
find ${DEFAULT_DIR} -type l to locate all symlinks under that directory; and then with each symlink:
column for formatting - making it easier to see the state of affairs and what needs to be doneWhile useful to be able to manually run to check existing/proposed configurations, really needed this to run before each backup, especially where settings may have been changed before a manual check could be run. Was able to force this in the following way:
/usr/local/var/urbackup/data/settings.cfg - improved the script to extract these if called without any arguments/etc/urbackup/prefilebackup so run before indexing/backupConstructing of filelist of "{client}" failed - TIMEOUT(1)/var/log/urbackup_client.log so that an admin can see what exclusions need to be introduced to get the backup working (since backup wouldn’t work anyway, without the necessary changes)Follow the links to see what it looks like.
Conclusion: The backup can proceed./etc/urbackup/prefilebackup without arguments, i.e. this would be run via. settings propagated to the client from the UrBackup server*/.wine*/dosdevices and */.wine*/drive_c/users/*/My Documents have been sufficient to cover the 3 problemsConclusion: The backup must be prevented until the infinite symlink loop(s) are resolved./etc/urbackup/prefilebackup with arguments for both DEFAULT_DIRS and EXCLUDE_FILES, i.e. running the script manually with test scenarios to see if exclusion rules are sufficient*/.wine*/dosdevices from the exclusion rules, to see what would happenAlso note:
SYMLINK (MAYBE EXCLUDE: LINK_TARGET outside of backup roots)/etc;Wine [https://www.winehq.org/] is a very common package installed by many Linux users. By default, it makes multiple symlinks as shown further above. Some of these will definitely cause infinite symlink loops, when backing up user home directories.
I have found that the following exclusion pattern pretty much works for all users’ .wine directories on a system:
*/.wine*/dosdevices;*/.wine*/drive_c/users/*/Desktop;*/.wine*/drive_c/users/*/My Documents;*/.wine*/drive_c/users/*/My Music;*/.wine*/drive_c/users/*/My Pictures;*/.wine*/drive_c/users/*/My Videos;
.wine* is to (optimistically) cover all of the wineprefixes [http://wiki.winehq.org/BottleSpec] that users will make - but this is only as good as users sticking to the .wine... directory naming prefix, for all of their wineprefixes!Ultimately, having to manually add these exceptions is going to lead to more users with problems. Really needs an automated solution of some kind.
With the current system, every change made to the configuration could arbitrarily introduce infinite symlink loops to backups. All it takes is shortening one of the paths in DEFAULT_DIRS or removing the wrong rule from EXCLUDE_FILES. At the very least, “outside” symlinks (see above) could lead to inappropriate directories being backed up - an undesirable outcome.
Having come from BackupPC myself, knowing that I had no such issues with symlinks when using it, I did a little research to understand how it dealt with symlinks.
[BackupPC Client Configuration - The Open Source Backup Wiki (Amanda, MySQL Backup, BackupPC)]:
$Conf{FtpFollowSymlinks}
This variable controls how BackupPC deals with symlinks. Default is false.
Linux/Unix
… In fact, by default samba makes symbolic links look like the file or directory that they point to, so you could get an infinite loop if a symbolic link points to the current or parent directory. If you really need to use Samba shares for linux/unix backups you should turn off the “follow symlinks” samba config setting. …)
Special files
Linux/unix file systems support several special file types: symbolic links, character and block device files, fifos (pipes) and unix-domain sockets. All except unix-domain sockets are supported by BackupPC (there’s no point in backing up or restoring unix-domain sockets since they only have meaning after a process creates them). Symbolic links are stored as a plain file whose contents are the contents of the link (not the file it points to). This file is compressed and pooled like any normal file. Character and block device files are also stored as plain files, whose contents are two integers separated by a comma; the numbers are the major and minor device number. These files are compressed and pooled like any normal file. Fifo files are stored as empty plain files (which are not pooled since they have zero size). In all cases, the original file type is stored in the attrib file so it can be correctly restored.
Me too! But with a bit more besides: as a default setting on Linux, I would prefer not to follow symlinks.
Personally, following symlinks is not a feature that is critical - since I wouldn’t be doing my job properly if I didn’t explicitly backup all of the important directories in the first place.
As for granularity, this might be much more complexity for very little gain. If the admin user defines the settings wrongly, he still isn’t immune to all of the potential banana skins. Not to mention the amount of headache involved in trying to work out these settings in the first place (for every single client?!) and then maintain them all - too much micro-management.
Replying to the above is a waste of time… uh, I mean to say that it will waste your time referencing the issue you’re replying to! So, in a bid to make it as easy as possible for anyone out there who cares to respond, perhaps a numbered list would make things a little easier:
@Uroni @mhohr @chungyan5 @AnyoneElse: Is there any benefit in sharing the script? If so, GitHub? Some other method?
@Uroni: Is there any issue with forcing a client restart to prevent the backup from taking place? Is there a better way of achieving the same goal of preventing the backup?
@Uroni: Following on from #2, I briefly looked through the source code but from my understanding, the prefilebackup is run but not checked for an exit code - with BackupPC, a non-zero exit code from the pre-backup script would prevent the backup from taking place - have I missed something here? Could this be something worth implementing, since it could be useful for a number of situations, if certain conditions are not in place at the time when the backup is going to run…?
@Uroni: Other than not following symlinks, is there a way of automatically recognising and avoiding the multiple symlink-trap that is found in all of the wineprefixes installed on a Linux system?
@Uroni: What do you think about using a symlink policy similar to how BackupPC operates? Symlink content vs. symlink target…
@Everyone: Not following symlinks by default for Linux: is this really, actually a better way?
Hello,
Any chance you could post the scripts you used? They would be helpful to me! Some of my Linux systems actually fill up the drive with “ERROR: Cannot stat “/usr/bin/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/rvim”: Too many levels of symbolic links (40)”
and I believe your script would save me lots of time!
Thank you in advance!
UrBackup client+server 2.0 has correct symlink handling. You might want to consider joining the beta ( UrBackup Server/Client/Restore 2.0.0 beta )
And since UrBackup server 1.4.9 you can turn off following symlinks in the advanced settings.
@Linxxs I came here to reply to you but @uroni got there first!
I’ve seen Settings > Advanced > Follow symbolic links on Linux but I haven’t got around to using it yet – good to know that this issue is now resolved, particularly looking forward to using v2.0.
Thanks for all your work @uroni.
thank you, I was so deep in trying to solve the problem, didn’t see that simple fix… I will try it and let you know.
Any chance 2.0 has rear integration? 
I am using urbackup for windows and i have to say it works perfectly! Thank you for the software in general!
Dear Mr uroni,
I have a UrBackup Server v. 2.0.4 beta and a UrBackup Client v. 2.0.4 on a Linux machine (Ubuntu).
I can’t unfollow symlink, on the client logs I have:
Following symbolic link at “/mnt/snapshot/home/stefan/.Private” to “/home/.ecryptfs/stefan/.Private” confirms symlink backup target “.symlink_” to “/”
How can I avoid follow symlink?
Best regards.
I just described it in this post here: UrBackup Server/Client 2.0.4 beta (with Mac OS X client); Restore 2.0.1
Hello,
After having tried bacula, amanda, CrashPlan without success, I tried urbackup and I am almost satisfied.
The only problem I had was on a Linux (Ubuntu) client connecting to a urbackup server on FreeNas.
The indexing was taking forever and was filling out all my partitions - with the indexing file and with the log file.
Finally I found out that the problem was my wine installation and the symlinks loop issue.
Disabling symlinks following that was enabled by default did solve my problem, and surprise! indexing took a few seconds instead of more than 24 hours as the previous time!
As suggested by someone in the previous messages, disabling the “follow symlinks” option by DEFAULT on linux would be very useful for newbies that never faced this problem.
I am using 1.4.9 for client.
Other options are, checking for .wine directories (and other well-known apps) and alert the user for the potential problem. Or checking for loops.
Just my 0.2 cents.
Thank you!
Best Regards
Federico
Or use 2.0.x where this is properly fixed :). Per default it follows symlinks only if the target is not already backed up.
Yes indeed, 2.0.x will be a solution, but if I am not wrong it is in beta status, so not for production, right?
Even if it follows the symlinks only if not already backed up, it will be stuck. Anyway this is not an issue if the user already knows of the problem 
Even a timer when indexing takes too long would work. I have a big disk (2 x 1T hd, + 0.5 T SSD) with a LOT of files (compiling yocto and android in many different flavours) but indexing had taken a few seconds when working.
Is there a way to make some directory exclusion on the client side, for example, putting a special named disk (like .no_urbackup) ? it would be very useful…
Thanks
Federico