Kinda hidden in that novella was the answer to your question. Think of each backup as a point-in-time copy of the files at that moment. When UrBackup manages storage by deleting older backups, it will also notice the files that are not needed for later backups (because of changes or deletion since then) and purge the no-longer-needed copies from storage. So yes, things can be lost, but not generally anything currently present on the sets being backed up.
Note also the special case of files that are present on several sets. These will only be deleted after all backups that refer to that version of the file have been deleted. If everyone has the 2021 Policy Manual but April is also keeping the 2019 version she worked on, UrBackup will have to keep both versions in storage, but only one copy of each. The 2020 Manual is no longer saved on any set, so as soon as the last old backup from 2020 is deleted, it will vanish as well.
tl;dr - For file restore purposes there is no difference inside UrBackup between ‘full’ and ‘incremental’ backups. Both contain the full file data as of the time of the backup
What if I’m NOT doing a “file restore” but rather a “full system restore”?
How does that change your statement?
While it is possible you might get a reply, adding to a topic when it’s already 2 years after the most recent prior post doesn’t give that high odds.
I still don’t understand what’s being said in this topic myself, as it violates what I know of both incremental and differential backups, neither of which should “work” at restore time unless there is a baseline full backup. I’m not seeing anything near to the amount of data backed up, file-size wise, in my incremental backups such that it would ever be able to use one to do a full system image restore were there no full system image to start with then apply the changes in each incremental backup taken since.
And when I say restore, I mean a “bare metal” restore on a system where the main drive that’s being backed up has failed entirely.
Well, you start talking about image backups while all above is about FILE backups soooo, there’s that.
Depending on what filesystem is being used incremental only backups are absolutely a thing where no full file backup is needed for fallback (after the absolute first one is done that then can be deleted when an incremental backup is made).
I do them and have been doing incremental only for a loooooooong time.
You’re right, if you accept that premise. But there’s nothing in the first post that indicates file versus system. It’s the follow-ups that presume file backups and it continues from there.
But, you’re 100% correct if what we’re talking about is file backups (except, perhaps, if something is deleted straight out of the shoot after a full and any one of the incrementals from which it could be restored gets nuked).
I did make my context clear, and I stand by what I said when it comes to full system image backups. If you don’t have the baseline full upon which the incrementals are applied, then the incrementals are useless.
In any backup scenario, file or full system image, I always follow a protocol where there’s an underlying full backup with however many incrementals until the next full gets taken. Then I may, or may not, depending on the circumstance, nuke the prior sequence of full and all associated incrementals.
Also note that the “revival post” asked specifically about full system image backups rather than file backups. You want a baseline full in that case, always.
When you make the ABOLUTE FIRST INCREMENTAL, yes, and nobody is claiming the opposite, not sure why you are arguing.
AFTER that it depends on stuff if it matters whether you can delete the full or not.
I once even tried to start with an incremental just to see how urbackup reacted, and it just made a full.
Since the question that got me into this thread was directly premised on how this process is different for full system image backups plus follow-up incrementals I stand by what I said. You cannot just toss away the full system image baseline and deal with only incrementals in that case.
I’ve been dealing with system backup protocols for decades now, and for system images, you either always take a full and keep it along with all its incrementals to do “bare metal recovery” or you just consistently take full system image backups at some reasonable interval so you can recover from a full image backup alone. It’s also faster to recover that way, though the backup itself is a lot slower. The reason incrementals came about in the context of full system image backing up is so speed of backup could be radically increased, but recovery time then suffers. There is no free lunch.
I think to clarify the question, Jill is asking for image backups, will a full image backup be deleted by UrBackup cleanup if there are newer incremental image backups that still reference back to it?
“You can select on which backup incremental image backups should be based. Either on the last full or incremental image backup or on the last full backup (sometimes called differential). Keep in mind that image backups can only be deleted if all the image backups they are based on are also deleted.”
This is because there is the traditional concept of full, incremental and differential backups - what you are familiar with - and then there is the urbackup concept of full and incremental backups.
THESE TWO CONCEPTS ARE DIFFERENT EVEN THOUGH THEY SHARE THE SAME NAMES!
The discussion below relates to file backups, not image backups. I have not delved into urbackups concepts related to image backups since I don’t routinely use images, although I’d guess that the concept is similar.
Note: This is also a discussion of HOW I UNDERSTAND URBACKUP CONCEPTS. I could be wrong! I don’t think I am, but keep in mind that I may not be stating everything perfectly.
I won’t go over the traditional concepts in detail because I think everybody probably already knows those. i.e., a traditional incremental backup will not work without all the other incrementals that came before it, back to the last full backup, which is required as well. To restore, you start with the latest full, then apply all the incrementals, in order, up to the latest incremental.
Say you did a full backup and it ended up containing 100 files. Then two files changed, and you did an incremental backup. The incremental would contain 2 files. To restore, you first restore the 100 files from the full. Then you restore from the incremental which will overwrite 2 of the files that were previously restored from the full. The end result being that you have restored the newest version of all 100 files (98 of them came from the full and 2 came from the incremental). This is the traditional concept. In this example the full contains 100 files, and the incremental contains 2 files.
But the above is not what urbackup does.
A full backup in urbackup’s concept means every file is copied to backup. Whether it was there before or not, whether it had changed or not. i.e., a “fresh” copy of the file is created in the backup. This is the same as the traditional concept of a full backup. So a urbackup full would contain 100 files. Same as traditional.
Now when urbackup does an incremental, it creates that containing the 2 changed files. BUT THEN IT LINKS THE 98 UNCHANGED FILES FROM THE FULL INTO THE INCREMENTAL. So the urbackup incremental will contain all 100 files. Two of those files will be newly written data, and 98 of them will be linked from the previous full backup. So with urbackup, BOTH the full AND the incremental will each contain 100 files. So you can delete the full and the incremental still hangs around WITH ALL 100 FILES. This is the difference between the traditional concept and the urbackup concept.
One might say, “Then why do we ever have to do a full backup?” And the answer is “You don’t!” You can just do nothing but incrementals. Including the very first backup you do. If you tell urbackup to do an incremental and there are no previous backups to compare against to see what files changed, then that first incremental IS a full backup.
So the next question one might ask is, “Then why does urbackup even give you the capability to do a full backup?” Well, there are some times when you might want to. Say you’ve been doing nothing but daily incrementals for 5 years. And say you’ve got a file that has never changed since the very first backup. So that file has been written to backup once, and now it has thousands of links to the same data from all the incrementals you have done since that first backup. Now say your hard disk storing the backups is getting old and corrupts a few bits on this file with multiple links to it. Now every single one of your backups contains this corrupted version of the file. Because all this time you only had one single copy of the file that was LINKED to many different incrementals. To mitigate this potential failure point you may want to do routine full backups every now and then. Because a full will FORCE this file - changed or not - to be re-written to backup, and it will be written to a different place on the backup disk than the original (potentially corrupted) copy of that same file is written to.
If you have perfect backup media that is guaranteed never to fail, ever, then there is no need to ever do a full backup in urbackup. However, since perfect backup media does not exist, you probably want to do a full backup once every couple of months, once a month, once a week or at whatever interval you are willing to risk your backups to - i.e., how long to you trust your backup disk to remain “perfect”? As your backup disk gets older and older you might want to protect yourself by doing more frequent full backups.
You are contradicting yourself.
No, you can NOT do an incremental without another backup to use as reference.
For fun, I even TRIED starting with an incremental on a new client using btrfs. It makes a full backup.
You HAVE TO create a reference, THEN you can do incrementals forever.
You can even remove the first full after an incremental because urbackup MOVES the files in the snapshot to the next snapshot in line when deleted.
An incremental BUILDS upon another backup, be it another incremental or a full, but you can NOT START with an incremental, that is a FULL. Be it that Urbackup does that FOR YOU so you don’t really notice, but it is STILL A FULL, not an incremental that is the absolute first backup.
This is exactly what I said. In the very lines from my post that you quoted.
You tell urbackup to do an incremental as the very first backup. The backup that is done will be a full. Regardless that you did not tell urbackup to do a full, you told it to do an incremental. It did a full anyway.
Maybe I could state it like this to clarify: “You never have to TELL urbackup to do a full backup” (because if it needs a full, it will do one anyway, even though you technically told it to do an incremental).
And where, pray tell, will those 98 links point if the source of said links no longer exists. Links do not contain data, they point to the location of the actual data.
If (and I’m not doubting it, but it’s unusual) urbackup does as @anon16079863 says - moving files from the initial full to “the most recent incremental taken after it” when an initial full file backup was taken, it is, in essence, keeping the initial full file backup (which is very, very unusual in the world of full followed by either incremental or differential backups). Urbackup essentially creates a “moving full backup baseline,” pushing the original full into the next available incremental, if it’s removed. It also sounds like it does the same thing if you were tl delete the “incremental that now functions as the full.” The files in that incremental that functions as the full would be pushed into the next incremental up the line.
Personally, even though this works, I don’t see the value in deviating from the classic practices in this way. It makes things more confusing, not less so, and that, in my opinion, is a very bad thing. It’s so much more clear to retain the “must have an initial full, known as full, and subsequent incrementals.” But I’m not the developer, so those are choices I don’t get to make.
You need to research the difference between hard links and symlinks. Symlinks (a.k.a. symbolic links) behave as you describe. Hard links do not. Urbackup uses hard links. Urbackup may use some symlinks for a few things - a symlink linking to the “most recent” backup would make sense. But for files that are part of a backup, hard links are used. Note that for some filesystem types - btrfs being one - urbackup uses snapshots and features of the file system in much the same way (conceptually) as it uses hard links in other file system types.
I used “simple language” for that statement. I didn’t want to expand to deep into the tecnicality of it. But yes, that statement is true, be it a snapshot that is relinked or hard/symlinks, it depends on the setup you created for urbackup.
The point is UrBackup is doing the heavy lifting in this, you don’t really have to care HOW (unless you are curious ofc)
What I get from that is “I don’t understand so it has to be a bad practice”.
You not understanding has VERY LITTLE to do with “best practice”. xD
And what you get from that is what you get from that.
Terminology surrounding backup and recovery and how it works is many decades old. There is a clear understanding of what full/baseline, incremental, and differential are SUPPOSED to mean in that arena.
I don’t give a flying rat’s ass how “superior” anyone thinks that UrBackup’s method (or any product’s method) is. When it confuses and conflates by muddling very well known and well understood terminology, it’s problematic.
And what’s been described here causes a real mess in the understanding of full/baseline versus incremental backups for most of the world that is doing them or has done them for a very, very long period of time. I think that’s a problem, you don’t.
It took you a while to finally get it so you are frustrated - that’s normal and I don’t mean to be disrespectful in any way - but you can’t just come here and talk in the name of most of the world
You are doing backups for many many years so I take you are aware of always incremental backup scheme. There is a nice post on Bareos site here. I wonder how it fits to your understanding of most of the backup world? What UrBackup does is not THAT much different, is it.
EDIT:
One more thing, related to this part:
It makes me less confident that you know always incremental scheme. Please spend 10 minutes on reading the Bareos link I provided, it describes (and illustrates) possible advantages very nicely - and then decide if it’s worth it for you or not.
I read that article and it’s informative. Note, and note well, that Bareos uses a specific type designation, “Always Incremental,” to describe what’s going on. It makes it very clear that “classic incremental” and “always incremental” for file backups are not one and the same thing.
There seems to be a misperception that I don’t understand what’s going on, nor why it’s advantageous, neither of which is true.
What I object to is that it is not, clearly, differentiated from “classic” incremental in UrBackup. It is clearly differentiated, both in documentation and syntax, in Bareos. That matters.
But with this I’m out, as that Bareos article makes clear, very clear, that the entire “always incremental” scheme is only appropriate when talking about file backups.
I originally entered this topic because someone asked about “how it would be different” were full system images being discussed. And you don’t ever do “always incremental” in that context. It’s still classic full followed by classic incremental, and if you were to delete the baseline full system image backup all incremental backups upon which it depends are no longer of any use.
This has always been an objection on using a specific and well-known terminology for a backup type that is not, in fact, that backup type. “Always incremental” actually involves mechanisms not characteristic of “classic incremental,” and that’s fine. But that “Always” makes it clear that something else is being done. Personally, I’d have called it something like “Roll-up Incremental,” but it really doesn’t matter what it’s called provided it’s clearly identified/differentiated from plain, vanilla incremental as it’s been understood for longer than I’ve been alive.
