Hi all, just a quick check with others, because im not 100% sure if this is indeed its just me again, but i had coincidentally mounted my arch rootfs on /mnt/tmp1 while on Debian, when restoring data to a different mount (/mnt/DUMP). Checking the log for the restore i see this:
After unmounting and attempting to recover another backupdir, i notice some weird errors:
Ok so it happened again. Because i had some unclean shutdowns while messing around (i know how that sounds, i know) with Arch, i run scrubs on my btrfs filesystems. All were good except for 2, one holded mostly a steam library on a nvme-backed lv, the other a lvm cached lv that is backed by 2 classic spinning rust disks, well of course the cache went bad. So for the first filesystem i simply recreated it, as it only holded a Steam library and i can simply reinstall the games. From the backups i had, i restored the steam compatibility data (proton/wine prefixes really) the screenshot in my earlier post reflects that (“Steam Runtime” and “Steam compatdata GamesNVME”, there i noticed the errors in the logs that i still wonder why those came from, as the files for which the errors are thrown had nothing to do with the restore attempt.
The other filesystem, with the lvm cache, is huge. From the journal entries that popped up during the btrfs scrub i was able to identify which files were corrupt. Only about 10 files. All except for 2 of them i had included in my backups. Again the files that i needed to recover that were left were related to steam. So again i attempted a restore (which btw all seem to succeed!) and i noticed during that icons from my taskbar were starting to vanish. I looked at first like untattended upgrades was upgrading some packages in the background. Then when i opened htop i could see some binaries that were running were not present anymore on the disk. Huh? So i decided to reboot. To find my system would not boot.
Fortunately i had re-setup Arch and from there with the help of the snapper snapshots i do run for the Debian rootfs i was able to sync back a snapshot from before the restore attempt. But at that time i did not think the restore was the cause. Until i got back in Debian and opened the urbackup webinterface and checked the restore log:
And it happened again. At this moment i am scared of using restore via the webinterface.
Im detailing the situation.
I have server_confirms enabled, so to initiate a restore i go onto the server webinterface, pick a backup, browse to the subfolder i need to restore and select restore, the restore action starts at that moment, without any interaction on the client side.
First i restored a subfolder on /media/DUMP. In particular /media/DUMP/Games/Blizzard/ which is in the “Games_DUMP” “backupdir” Everything went fine, no issues reported in the log on the server side. Nothing at all mentioned at the client log.
Then i needed to restore “/media/gamesnvme/SteamRunTime/steamapps/compatdata/4230314876/” which is part of the “Steam Runtime” “backupdir”. As i was waiting for the restore i was watching a youtube video from level1techs and had the folder opened in dolphin on the other monitor (on the same client that the restore was running on). Then Dolphin just closed, and i tried to reopen it. An errormessage appeared at the right bottom, saying the Dolphin executable existed, but didnt have the execute bit set (what??) then the youtube video hang, looping a couple of seconds of the audio constantly. The machine was completely frozen (ctrl-alt-f2, -del, no response) and i had to hard-reset. Grub greeted me and after that a kernel panic. Used Finnix to restore a snapshot (basically rsync -avHXp /snapshot /rootfs) to restore a working state of my rootfs. After that it booted.
The moment it froze:
Is part of the urbackup restore to “walk through” the data of the backup that is not relevant but somehow it restores that irrelevant data in my case? Looks like it to me. Like say you include 2 backupdirs in a backup FolderA and FolderB, for (obviously) different non-nested paths. Then when trying to restore FolderB, it walks through the backup archive until it reaches the contents of FolderB, but in my case, for some reason, it also restores the contents of FolderA. But im just guessing ofcourse.
I might run some tests this weekend like restoring from the commandline to a different path, to see if it somehow creates the files/paths that should not be part of that restore, but insightful responses are very welcome!
EDIT: it seems the photos i took with my phone are not displayed and i tried to re-insert them without success. If relevant ill look into that later.
You probably have some symlinks to the root directory. E.g. wine has those.
When restoring with the command line you can use --no-follow-symlinks to not follow symlinks during restore. You can also change the backups to not follow symlinks.
I think it might be best to change it to not follow symlinks when starting the restore from the web interface…
Sorry about the delay Martin, i have not needed to perform restores in the meantime and as you can imagine i wasnt too happy about the situation so i was content with knowing the date was being backed up at least, the restore is for another time.
I noticed in the client changelog: “Disable following symlinks by default when restoring via command line“, could that be related to this thread? I noticed all clients have auto-upgraded their client, good. I could still run some tests with the suggestions you made in June, if you want? I assume the behavior is still different when restoring via the webinterface vs the command line client? What i dont understand is that i have never seen the option to not follow symlinks during a restore via the web interface, where can i find that? Or maybe you meant to also change that default behavior (not a user interaction, but “under the hood”)? The whole problem this post is about feels like “symlinkception” hehe. Thanks ofcourse for your work and help, it is much appreciated. Let me know what tests i can run to help/confirm things.
(bit offtopic, bit related) Maybe you remember i also made a post about symlinks not always being followed when follow_symlinks was enabled? I ran into another issue where the modification timestamp on a lot of folders was changed on a client and now the backup server is transferring and comparing (and linking unchanged files) all the data, about 10TB!, which will take days. Its not a big issue, and the behavior is not so weird, i assume it comes from the fact i am not using snapshots on that client. So i had another reason to start using snapshots. Right now i am trying to figure out how to change a client to use (btrfs) snapshots for backups post-install, that is after the client was already installed with no option for snapshotting chosen during installation. What i found so far is that if i uninstall the client (uninstall_urbackup), then install the client again (this time with option 3), edit /etc/default/urbackupclient, then run urbackupclientctl set-settings –authkey “<client_id>“ –server-url “urbackup://yadayada” and restart the client daemon, initially the server complained that it was recognized as a new client with the same name, but as i was writing this down that changed and the client appears as normal on the webinterface. Comparing the snapper snapshots taken around the reinstallation of the client, i noticed a file “/usr/local/etc/urbackup/no_filesystem_snapshot“ was removed and “/usr/local/etc/urbackup/snapshot.cfg“ was created with in there references to the “btrfs_create_filesystem_snapshot” and “remove” scripts.
Could i simply make those changes on existing clients and assume btrfs snapshotting will be attempted in the future or do i need to uninstall and reinstall their clients? That would be really useful knowledge. Im yet to test the client to see if it works as expected, but ofcourse now with snapshots. I’ll be sure to update the other post with my findings. I was wondering if you could also check the links to the wiki and “other documentation”, i dont see a way to share/consult information at those places other than the admin manual and this forum at the moment. Cheers, Daan.
I wish i remembered this undocumented parameter, i just recovered from another instance where i tried to restore a steam related folder (with proton/wine prefixes in it) and the same thing happened again. I was using the command line now, urbackupclientctl AND i was using the --map-from and --map-to options. With those map options i was attempting to restore the files to a subfolder under /tmp. Basically the same symptoms; fonts in the browser were off, icons from the taskbar started dissapearing. But there was a difference: when i saw that happen, in the konsole where i had a sudo -i opened, and did a “ls -l /”, i dont remember the exact error, but it looked something like I/O Error and nothing was listed (and it dawned to me the same thing happened again). But when the restore finished (i still had the webgui open) things started working again, but i immediately noticed something off: instead of /tmp having root:root ownership and 777+t, it was owned by my useraccount and 775…weird! Also /tmp is a seperate mount, in case that matters. While the restore was ongoing i wasnt able to do much, i tried to take a screenshot of the exact command, but ofcourse the snipping tool wasnt working at that exact moment because my system was being butchered.
I had made previous attempts at restoring via the command line, using the --map-from and --map-to parameters and needed a couple of tries before i was successful because the mappings didnt work, it would just restore to the original location. No errors were thrown and no way to cancel with any of those attempts which didnt help with my tries.
Thankfully i started trying on a folder that i knew was small, to make testing quick. There does not seem to be a way to cancel an ongoing restore. I might do a ctrl-c, but the webgui shows the restore is continuing, i can see files are modified in /tmp too and the target folder appears to be restored but permissions are only set at the last stage. Also on the webgui there is no cancel button. When i finally figured out the --map-to/from parameters take ABSOLUTE paths, rather than what i assumed relative paths to the backup folder, i started restoring the steam related folder and everything went to shit. Here is a screenshot of what i remember to be a journalctl --since -1h --unit urbackupclientbackend --grep ERROR:
We really need some documentation on that urbackupclientctl tool, some parameters are unclear, the terminology is confusing, e.g. when specifying a a backup folder with urbackupclientctl add-backupdir and remove-backupdir, the -d parameter is used to specify a path, urbackupclientctl restore-start --help says it is used to specify a path, but the -d parameter is used for the NAME of the backup component. Also the --follow-symlinks parameter exists, but this parameter --no-follow-symlinks is not even mentioned. It doesnt say that even when --follow-symlinks is NOT used, it will still follow them by default (i find that last one really strange behavior).
I will now attempt to restore the folders using the explicit --no-follow-symlinks, i really hope it works. If it doesnt, i will need recover again and simply sftp the files over to my machine and change the perms. Thank god for snapper.
It appears symlinks have not been followed anyway. I have reported that earlier when that was an issue, here its actually better. I want the symlinks, just like they appeared on the original location. But the symlinks have been re-written, presumably to save resources at the server end. Also this was found earlier. But now that is a problem, because not only are permissions broken, so are the links. Note that if i restore such an individual link through urbackup, it is restored with the proper permissions and pointing to the proper resource. The problem is ofcourse, there are tons of symlinks inside the prefixes, which are now all broken.
In the meantime i have spent hours attempting to recover the data, finally it looks like it has been succesful; basically i did what was described here, although a bit different. I had a template Debian 12 machine ready and cloned that, used virt-customize to change its hostname and inject the client installer for my desktop, and added a significant disk to the vm to mount at the target path (all under /media/gamesnvme). Stopped urbackupclientbackend on my desktop, booted the vm but forgot it already had the client installed. Apparantly urbackupsrv seen it appear on the network momentarily, and now it seems the name for the template i had changed to “recover”, the hostname i gave the new vm. I suppose that will change back the moment i bring back online the template, but in any case its not a biggie. I did have many issues starting the restore though, i kept running into this until i decided to change the settings to “server-confirms” and just try via the web gui. And guess what? The restore seems to happen without messing up the virtual machine. Lateron i will mount the vm’s qcow locally via qemu-nbd and copy back the data, finally!
So all in all, this --no-follow-symlinks does not appear to exist, maybe you are developing a new client and mixed things up? I would really love to see one would be able to restore data for another host, instead of having to “impersonate” the original machine.
