How to keep UrBackup working?

Server
1

I have been using UrBackup Server (2.5.32) on Linux with Windows 11, 10 and Linux clients. Once I do a clean install it works fine. But when I have to reboot the server, I try to stop any backups running of course it will not stop them. I try to wait until it finishes. If something happens the server goes down UrBackup is toast. If you reboot the server Urbackup is toast, very few times I can reboot and it will not destroy UrBackup server. I have never been able to get the system to run right again after reboots. I end up having to reinstall.

Next issue, Datto on linux. It constantly leaves snapshots behind and fills up the filesystem. I can’t seem to find any documentation on how to remove them without trashing the filesystem then nothing works again.

Any ideas on how to keep this working so I don’t have to delete all the backups and start over each time would be great. The product is great when it works, but when it gets confused the only way to fix it is delete it all and start again. It would be nice to have documents that cover basic repair and cleanup. That is not a good solution when you have backups you need to keep. Is there a way to import the backups if you have to reinstall? There has to be a better solution than reinstall!

Thanks.

2

That is what you leave us with trying to figure out what is wrong.

No, this is not good enough.
How is it installed.
On what system exactly.
Versions.
Logs.

There are so many things you could have included.

3

Sorry, I was just looking for more information as to way to fix datto and upgrade issues that break the box. I went into more details just for completeness.

Issue Overview:

I’m encountering recurring issues with UrBackup on my Ubuntu 22.04.03 server where it will not backup any clients anymore. I use BTRFS for storage at /media/backup/urbackup, and the problems seem to appear after updating UrBackup most of the time or rebooting the server it will randomly happen (so I try to never reboot). Despite my best efforts, I haven’t been able to find a permanent solution to the issue.

Observations when this happens where no client backups complete:

Windows Clients (e.g., Windows 11):

  • I often encounter messages like:
    • INFO: Referencing snapshot on Windows11c for path D failed: FAILED
    • ERROR: Error getting complete file VaB8Em7VZXRMW93Ctv9yD from Windows11c. Error code: CANNOT_OPEN_FILE (3)
  • Windows clients appear to get stuck during the backup process, and even new clients show the same behavior. Upgrading UrBackup via the interface sometimes leads to issues with older backups not being visible to clients.

Windows clients will sit at: 0% 0 bytes / 562.59 MB and give an ETA but never do anything.

Linux Clients:

  • I’ve noticed the following error message: Referencing snapshot on "LinuxC" for path "rootfs" failed: FAILED
  • Linux clients are running UrBackup v2.5.25.0, and I’m having difficulties completing backups with them.

Linux clients will sit at: 0% 0 bytes / 163.77 GB and no ETA but never do anything.

All clients
Any backups that say they complete when it is like this don’t really complete they show Used storage/0 bytes But doesn’t give any errors or information as to why it didn’t backup anything.

Specific Questions:

  1. Is there any documentation or best practices for upgrading UrBackup without breaking the server?
  2. How can I fix the server when these issues arise, preferably without reinstalling everything?
  3. Is there a way to retain existing backups during reinstall?
  4. Is there a way for clients to automatically reconnect without requiring manual intervention after a reinstall?
  5. Regarding the Datto issue, how can I resolve it when snapshots are not cleaned up automatically so it doesn’t require a complete rebuild of datto and Urbackup?
  6. Is there a way prefered way to backup the configurations and not the client backup files just to help speed up reinstalling?

I’m eager to find solutions or workarounds to these recurring challenges with UrBackup. Any assistance or guidance is greatly appreciated.

4

Sorry I cannot help you with most of your problems but I also got a lot of errors since my upgrade to Ubuntu 22.04 and the latest urbackup version. I don’t really know what is going on yet but it seems to me as if datto is not working properly. When I manually create images via datto they sometimes cannot be mounted which is why I for now disabled snapshots with datto completely. After disabling datto my Linux Backups run smoothly, before that I had lots of errors like „cannot get complete file“ and the like.

Regarding the datto cleanup:
I created a small script for cleanup. You may want to have a look at this and if necessary adapt it to your needs. You can find the script here:

1 Like
5

Yeah, I found your script (thanks) earlier today and tried that, that for clearing the snapshots not cleaned up, but it seems the issue is on the server side. I can’t get any clients to backup. They all just stay at 0%.

6

Then I suggest you follow his advice and try if that fixes things.
Stop using img backup (or at least datto) and see if it changes anything.

And you tested it like the documentation describes?
urbackup_snapshot_helper test reports no errors?

https://www.urbackup.org/administration_manual.html#x1-11200011.7.2

What do you mean by that?
What is “upgrading via the interface”? Do you mean CLI?

It seems that on ubuntu you add the repository and apt should keep it updated.
(I have never used it but that is what the documentation tells me)
So: sudo apt update && sudo apt upgrade to update it.

What errors do you get when updating?
Logs?

https://www.urbackup.org/administration_manual.html#x1-60002.1.2

Yes, on ubuntu you seem to use apt.

Again, I don’t think you should “reinstall”, I think you should apt upgrade.
But on Debian, I have to “reinstall” every time, and that should NOT overwrite anything, at least it never has for me.

Same like above, should not change anything.

Use the great little script @DrRSatzteil provided. Although, that first cat $FILE? xD
(not sure you need to cat that, you are not changing anything like with the cat:s further below.)

Again, you should not need to, but iirc there is a setting in the web gui that tells urbackup to backup settings in the nightly cleanup.
Otherwise, on Debian, /var/urbackup contains the settings, not sure about ubuntu but probably same location.

My personal opinion, forget img backups and use file backups only.

7

Hmm as far as I can see I’m writing the content of $FILE to the variable SNAP_COWFILE_PATH that I’m referencing on later in the code to delete this file. Or am I missing something?

EDIT: Ok I see. You are right it’s not really needed… well it doesn’t hurt for now either :sweat_smile:

1 Like
8

Hahaha, after just one sentence the coin dropped. :stuck_out_tongue:

It makes no difference at all, I just found it funny to comment on.

Thank you for doing stuff like that script! :heart: :heart:

1 Like
9

Apt is updated and upgraded. I am not really doing images. I do them on windows, but not on LInux.
The linux image comes at too large a cost. You have to have more than double the space to build. I
think the apt upgrade is what started all of this.

I checked BTRFS and it seems fine from the checks. So must be in UrBackup that it is having issues.

Errors 10/17/23 09:20 Error getting complete file "VaB8Em7VZXRMW93Ctv9y D" from (ACER). Errorcode: CANNOT_OPEN_FILE (3)
Errors 10/17/23 09:20 Error getting complete file "VaB8Em7VZXRMW93Ctv9y C" from (ACER). Errorcode: CANNOT_OPEN_FILE (3)

It seems it can’t open any files. I don’t think it is on the client side, because even new clients have the same issue. Backups have been down long enough I am going to have to just start over again. I really wish there was a way to repair the system when it gets in this state and not have to lose all the backups. But as far as I can tell when it updated something broke in the BTFS filesystem (Not a true BTRFS issue but how UrBackup saves files) and it couldn’t open the files anymore. Permissions are all correct all files owned by urbackup.

Doesn’t help that I had the dreaded Ubuntu UEFI bug that wouldn’t allow it to reboot just lands in Grub rescue. Fixed that by applied the Ubuntu patch and updating grub.

Any other ideas before I blow it away and start over?

10

Nope, but if you run the urbackup snapshot helper I posted about earlier, it would tell you if it is related to btrfs at all or not.

If you want to keep your settings and database, just back them up and make sure to put them where needed after you reinstall. Just make sure urbackup is not running the nightly cleanup before putting the db back or all btrfs snapshots might get removed.

11

Yes, tested that and the with btrfs to ensure the volume was ok.

Thanks for trying.

12

Well, if urbackup could create a snapshot with the helper, the error is most likely not on the server side.

13

What else could it be? All clients worked then they didn’t. Even new clients don’t work. Only thing that changed was update and a reboot.

14

Strange. I was reinstalling and was going to try to keep the old data. After reinstalling the OS I mounted the btrfs filesystem and went to change the permissions of all the files. It seems that all the clients had files marked as read-only. But I could write files to the BTRFS filesystem with no issues.

Is there an chattr permission being set on the files? Is this maybe the issue that keeps showing up after reinstall. This time I went with Debian and didn’t install with apt. I decided I would rather like to have control over when it upgrades to ensure it doesn’t break the system. But it seems it breaks something. I have had this happen each time I upgrade.

15

If you changed operating system completely and the filesystem STILL acts weird, it’s pretty safe to say it is outside of what urbackup is doing.

Are you SURE you can actually write files to the system and it doesn’t just “seem like it”. Have you rebooted and the files you create are still there?

A btrfs in r/o is a clear sign that something is not ok (if that is the case here).

You have to provide way more info if you need someone to be able to help out.

16

Yes, I tested with a reboot. I did another after BTRFS test and after I found the files that said read-only, it was not all files but every client had them and the new client I added did too. I figure If someone else has the issue at least they will have somewhere to start. If well more of when I have it again I will start down that path first. But for now I don’t plan to upgrade anymore without some major planning due to losing all my backups each time I do. I installed from source this time too instead of using the apt. So far everything is running great 989Mbs. It seems to perform way better when I build from Github code. I just couldn’t play with it anymore to track it down. I needed to get good backups so I could sleep well at night ;).