Understanding the Principles of Backups

general discussion
1

Hello,

First I want to say, that I am new to the whole server space, backups, everything that comes with it. Sorry, and thank you for your time and patience.
I want to get a better understanding of UrBackup and how a backup strategy plan should be made.
I have read the manual, all documentation, and multiple forum posts and cant seem to answer my questions or expand my understanding.
I have lost many files over the years and am trying not to repeat.

Use Case:
I have 3 window’s laptops (win 10/11) with roughly 500GB each for myself and two other clients.
I would like to backup the data to my server, which is running Unraid, with Urbackup in a docker container. I want to keep all my files, even if the user deletes it from the laptop/client side, unless I go to the server and remove the file myself. (note file, and not image, as in it could be a document, picture, folder, etc.)

To my understanding:
Urbackup can create Image backups, which are complete volume backups of windows, and file backups, which are backups of the files a client can specify.
Urbackup does a full backup, which basically copies everything, and then incremental backups, which watches for changes and only backs up the changes or new files if it finds any.

Incremental backups use symlinks, which is a link to a folder on the full backup, but sometimes it’ll use hardlinks if its a new file? Symlinks break if the original starting file is deleted. In my mind I picture a single rope connecting to one file which connects to another in a line, if the anchor breaks you lose all the links, while hardlinks are multiple ropes connecting to a single file, where if one anchor is lost, you still have the others. Which is also why Hardlinks show each hardlink is 10GB for example, even though its only a single 10GB file, because they get the file size from the source. Where as symlinks can show small incremental sizes as they are all tied.

Full backups and incremental backups are eventually deleted to make room for newer ones, based on the number of copies to keep, which are configured in settings.

The multiple parts I am confused about:
If a client deletes a file from their laptop, the file remains on the server in a full or incremental backup. However, when the full or incremental backup is eventually deleted and replaced, will the file also be deleted off the server? (This is not what I want, as I always want to retain all files ever sent to the server. I don’t need versioning, but it would be a bonus for documents since they can be edited)

On the server, the files are stored under a .directory_pool folder, however going into it, there is no structure as there was when backing up. All files are stored in separate folders labeled 2H, 77, 8G etc. This is not readable, so I thought user views would provide the same file structure using hard/sym links, however this folder is empty. Unless viewed through terminal where you can see symlinks. How can this be viewed properly, similar to how it was backed up from the client?

If I was to delete a file that is no longer needed, or is taking up to much room from the .directoy_pool, will it corrupt the database? and how would you even find the file as its not easily viewable? There doesn’t seem to be a way to delete a file from the UI, only the complete incremental or full image.

Thank you for your assistance! … Sorry about the essay :stuck_out_tongue:

2

I’ll address just this point, as it’s not specific to UrBackup, per se, but backing up in general.

Yes, the file will be deleted from the server. The conventional concept behind a backup is to keep what amounts to an exact copy of how the source machine (client, in this case) is at a given moment in time. That’s whether you’re doing file backups or full system images.

You can think of the initial full file backup as a baseline, with incremental backups making notes about what files that were in the baseline backup were changed or deleted, as well as what files came into existence after that baseline backup was made. Were you to run a full restore with ALL of the incremental backups used, you should end up with a collection of files exactly as they were as of the moment of completion of that final incremental backup. That means that any files deleted somewhere along the way will be deleted, new files that were created will come along, and any files that existed at the start, but were changed one or more times will be in the state that they were in at the time the last incremental backup was taken. I consider a full backup to be “the baseline” and when any given full backup gets deleted all incremental backups related to it need to go away, too, as there’s nothing left to apply them to. The next full backup, which will have been taken prior to nuking an earlier full backup and the incrementals that would apply to it, then becomes the baseline. But the end goal of backup software is to re-create exactly what exists as of the moment a backup is kicked off. It is possible, of course, to use one full backup and only some of the earlier incremental backups, not progressing all the way to the last incremental backup, but in order to do that and get what you’re looking for you’d have to know when a given file was deleted.

What you’re looking for was once provided very tidily by Microsoft SyncToy (which was not a toy by any means). It allowed you to set up your source and your destination such that one of three things happened in the destination as changes happened to the source. You could have synchronization, where changes in the source are made in the destination and vice versa. You could have echoing, where new and updated files in the source are copied to the destination, and any renames or deletes in the destination are also done in the source. And, finally, you could have contribution, where new and updated files in the source are sent to the destination, renames in the destination would be repeated in the source (which isn’t a factor for you), but no deletions done in the source were ever done in the destination (unless you did it by hand). In a contribute situation the destination became an ongoing repository for the source where active files were backed up as needed, but if a file was deleted from the source side the last version of it that was contributed to the destination side was left there forever.

What you describe wanting can now be achieved using the ROBOCOPY command in Windows with the /XX switch, which excludes extra files and directories present in the destination but not the source. Excluding extra files won’t delete files from the destination. If I’ve read what you want correctly, it is to exclude any files that have been deleted in the source from being deleted in the destination. There are scads of tutorials on how to use ROBOCOPY to achieve the ends you wish to achieve with regard to file backups, and you might want to consider it for that purpose.

I’ll leave it to others to get into the UrBackup internals, as I cannot speak to those.

1 Like
3

@britechguy, Thank you so much for your detailed and knowledgeable response. This really opened my eyes up to how things work.
Currently I use Nextcloud to back up my files and the other clients who are using android phones, but its a bit clunky to use for windows, which is why I went with Urbackup. I have read many instances of DB corruption using nextcloud for large windows backups, so I will definitely look into RoboCopy. Thank you for the suggestion.
I guess I’ll continue to use Urbackup for full images. I could just increase the increment amount, and use more full backups. Then using archives, build out a retention policy. This way I can mount the full backup to get my files back in that moment, or restore the laptop fully using restore. But for the file backup, and to maintain all files ever, I’ll look into Robocopy. Thank you again.

Anyone else want to chime in for the other two questions, it would help. :pray:

4

It sounds like what you want, were you to use UrBackup alone, is to do NO incremental backups at all, do full backups (whether file or system image), and then come up with a retention system for the series of full backups that meets your needs.

I still don’t see this working as far as file backups go because they will reflect deletions, and once your oldest retained full backup gets aged off, if there was something deleted prior to the “next youngest” full backup, it’s gone when the last full backup that contained it is gone.

You’re not really trying to have a conventional full file backup protocol based on your own description. It’s more of an archiving protocol, with any files actively being used being updated in that archive, but any files deleted from the source not being touched in the archive.

I get what you’re looking for, but conventional backup software is not the best way to achieve that end. There are other tools, and readily available ones, better suited for doing what you want.

5

Yes, you are exactly correct. I just meant I could use Urbackup for image backups so I can restore one of the devices if ever needed.

But yes, that’s what I am looking for. Not a conventional backup solution, but more so archiving like you mentioned. Basically what you mentioned with SyncToy, and how it could send the files to a source, keep tack and update them, and wont remove the files if they get deleted from the source.

I’m very new to the space, so I have been trying to find better solutions but keep coming up empty. If you can suggest some other solutions, that are free, or open source, or even charge a one time fee, that I can run in docker, it would be greatly appreciated. Thanks again!

6

I am not quite certain what you mean by “can run in docker.”

I have given you one solution from the standard Windows command toolbox already: Robocopy.

That being said, since you mention versioning, if these archives are meant to be on a per Windows machine basis, you had ought to look at File History. It’s a bit fiddly to restore from, since it does keep file versions, it’s dead simple to set up for a given machine. The one thing I tell everyone to do is to NOT keep the version history set to “forever.” I’ve never seen anyone need a version of a file that’s 10 years old. If retention is set to 6 months, it will keep the most recent 6 months of versions until the file stops being changed. Then as time marches on it will start tossing the oldest version (as in it would be 6 months plus one day old) each day until you were down to the single most recent version, which will be retained until you delete it. It’s possible to make File History also keep files in its archive that have been deleted from the original source. Look at the documentation regarding settings.

7

That’s great. However does File history work similar to how urbackups file backups work?
As in, is it more like a backup or can I use this for archiving and not have any files deleted from the destination?
It looks very easy to setup compared to Robocopy, but is it just copying files over, or is it watching for changes and incrementally updating them and not creating large duplicates?

8

File History is a fully versioned file backup (not system image, just data files) solution for a single machine.

I strongly suggest you do some web searches on “how to use File History” as the resources out there are almost endless, and the explanations of how to set up and how to tweak to meet your needs are good.

This is a case where I cannot possibly explain File History better than Microsoft and a very great many third-parties have already done.

9

I have setup up windows file history and been playing with it for a few days now, but am experiencing so many issues with it. First it completely bogs down the laptop while tryin to set it up, or settings crash. It copies files that I have said not to. Its very limited in what it can do, and requires the parent folder not be excluded, even though there are other folders you don’t want. Its very tedious to setup and you have to be very granular and exclude everything you don’t want, rather than just saying, “copy only this”. There seems to be a $OF folder created, where half my files are randomly stored in thousands of folders. This seems to be a “overflow” folder designed to combat long names possibly. The app crashes when trying to clean up old backups, and the minimum logs it does generate seem to be useless. Just seems overall a hassle and unreliable to use. Used Robocopy as well, and holy cow is that tool fast and powerful, just wish it had more features like backup tools do.

10

You’ve got to figure out what works for you, hence the reason I’m recommending commonly used tools that come with Windows.

I have not heard of the issues you’re having with File History with the exception of the rare “file name too long” class error (which happens with other tools, too). Of course the initial File History backup on any machine with lots of files on it is going to take lots of time. It’s fetching each file, copying it, updating the initial version index, etc., etc., etc. Versioned backups are neither fast, nor simple.

With regard to ROBOCOPY, I have no idea what you mean by “wish it had more features.” It’s got more features than virtually anyone can use. The number of switches and how they can be combined makes it “the Swiss Army Knife” of file backup software. I set it up to Mirror (which you don’t want, at least not in the conventional sense) a source to a destination in mere moments. If you can’t find a way to get ROBOCOPY doing exactly what you say you want, then it likely can’t be done. This utility has been a backbone of file backup for many, many years now.

1 Like
11

You might also want to have a look at Restic.

Given what you want, you’re almost certainly going to have to tweak default behavior to prevent deleted items from disappearing, but I believe that can be done.

12

Try duplicati, it uses deduplication and you store only the changed blocks, so you can do backups forever. :slight_smile:

13

Thank you both!
I tried Restic, but I need to look more into it. It is a little above my ability and understanding currently.

I have looked into duplicati, but again I would run into that issue of the backups overwriting each other eventually, and would lose the data that may have been deleted from the source. Also during my research, a very large amount of posts all said to avoid duplicati because it struggles with recovering large images. Might be a good idea to check your back ups and see if you can restore them.

14

There is an old saying in the IT world: “A backup isn’t, unless it can be restored.”

While it is impossible to test each and every backup you might make once you have a protocol in place with a given product, you definitely need to attempt to restore when you are first putting that protocol into place to make sure you know how to do so and to see that your backup is not corrupted.

I have not, as yet, attempted a restore from UrBackup as I’m in the early stages of working with it. But I would never even think of deploying it without first having attempted a restore on one of my testing machines before doing so. And that’s true regardless of the product, not just UrBackup.

@Deathmoney, in the end, you are really not looking for a backup solution in the conventional sense, but an archiving solution. As I’ve said before, and will reiterate here: The intention of a backup is to allow a system (or collection of files) to be recovered/restored to exactly the state they were in when the backup was taken. By its very nature, that means that as files get created and deleted in the source, they should be created and deleted in the backups of that source. You are looking for a one-way system where everything ever created gets added, everything that was changed gets changed, but nothing that gets deleted gets deleted. And if what you’re trying to handle that way is a collection of files, I stand by ROBOCOPY as the way to achieve that end.

15

Yes, thank you!

Yes, I am looking more so for a one way archiving tool. I have been using Robocopy, and so far its working great. It can be slow, but I think I just need to play around with the options more.

The reason I wanted a archive solution, is because I wanted everything stored in one location, which is my server. From there I am creating a true backup of the contents on the server to an external drive, and to the cloud once I figure out how and who the best provider is for the money.

You enlightened me on backups and how they work, and their use case. This is perfect for step two of my strategy. I was hoping for a GUI app to do the first step, which is the archiving. Robocopy is currently doing a good job of step one, once I figure out how to automate it, it will be the key to my strategy. Thank you again for your help!

16

To restore files go to the Urbackup server interface at http://Server-IP:55414.
On the “Backups” page, select the desired client and drill down to find the file or files you wish to restore.
Drill down to a single file, left-click it and it will typically be saved to your downloads folder.
You also have “Download folder to ZIP” and “Restore folder to client”.

Yes. So don’t remove files from the server.
Files, once backed up, will remain on the server forever, until removed from the server either manually or automatically.

Yes.
But that is fixable, but not desired.
Running remove_unknown.bat will shut down your server, completely remove all database records that point to missing real files and completely remove all real files that have no database record.
This is not desired because the server will be offline for hours to days while remove_unknown.bat is running.

They don’t have to be.
You can disable full file backups to do only incremental file backups.
You may set your incremental frequency as desired. 1/hour, 1/day, etc…
The first incremental will be a full file backup because there is no prior incremental.
Each incremental file backup therefrom will backup only new or changed files.
You may set “Maximal number of incremental file backups” to something like 1,000,000.
That will keep incremental backups essentially forever.
You may set “Minimal number of incremental file backups” to something like 990,000.
Are you starting to get the idea?

Give Urbackup enough storage space to save all of the file changes expected for as many clients expected.
TrueNAS can host a Urbackup server and easily, not cheaply, provide 100’s of terabytes of storage.
TrueNAS and ZFS are also reliable and repairable enough to support Urbackup for many years to come in the face of hardware and/or drive failures/changes/upgrades and, whatever computer technology may look like decades into the future.

If you need a file that has been deleted from a client.
Drill down in the server’s “Backups” page to an incremental backup dated on or before the deletion date and restore the file using one of the various methods I outlined above.
Deleting a file on a client will cause that file to disappear from all future incremental backups but, deletion on a client will not remove the file from any incremental backup made before the deletion date.
Making one incremental backup every hour will take more than 114 years to make one million incremental backups.

Regardless of what is being done with incremental file backups, Image backups may also be performed on an entirely different schedule in order to provide bare metal restore if a C drive lets the smoke out.

Also consider running two servers.
Each client may be set to backup to both servers.
Each server stands alone. One server cannot backup the other.
If a server is lost, the other server keeps on keepin’ on while the lost server is fixed.

I hope this helps.

All for now

17

I mentioned this earlier, too, but it’s a grand PITA if you have no idea of exactly when the file was deleted.

I don’t like UrBackup for that intended purpose. Something like Windows File History or other versioned file backup schemes make it far easier to pull out long deleted files provided you know the names of the files you’re trying to retrieve.

It’s a “tool to task” thing, for me. Versioned file backups with archiving and backups such as those taken by UrBackup aren’t for the same purposes.

18

Hey britechguy

Simply go to the restore interface as described above or through a right-click of the client tray icon.
Go to an earlier backup that you are pretty much assured that the file exists under.
Not so early that the file has not been created yet and not so late that the file has already been deleted.
Once you do find the file in any one of the incremental backups, like this image below;

List
List1180×531 22.2 KB

click the list button next to the file. (I selected desktop.ini for this example)
You will be presented with a page similar to this image;

Listing
Listing1179×819 34.8 KB

which lists every incremental backup that contains that specific file.

Not so difficult after all.

All for now

19

Thanks very much. Another instance where a picture and brief explanation is worth a thousand words!