Since the question that got me into this thread was directly premised on how this process is different for full system image backups plus follow-up incrementals I stand by what I said. You cannot just toss away the full system image baseline and deal with only incrementals in that case.
I’ve been dealing with system backup protocols for decades now, and for system images, you either always take a full and keep it along with all its incrementals to do “bare metal recovery” or you just consistently take full system image backups at some reasonable interval so you can recover from a full image backup alone. It’s also faster to recover that way, though the backup itself is a lot slower. The reason incrementals came about in the context of full system image backing up is so speed of backup could be radically increased, but recovery time then suffers. There is no free lunch.
I think to clarify the question, Jill is asking for image backups, will a full image backup be deleted by UrBackup cleanup if there are newer incremental image backups that still reference back to it?
“You can select on which backup incremental image backups should be based. Either on the last full or incremental image backup or on the last full backup (sometimes called differential). Keep in mind that image backups can only be deleted if all the image backups they are based on are also deleted.”
This is because there is the traditional concept of full, incremental and differential backups - what you are familiar with - and then there is the urbackup concept of full and incremental backups.
THESE TWO CONCEPTS ARE DIFFERENT EVEN THOUGH THEY SHARE THE SAME NAMES!
The discussion below relates to file backups, not image backups. I have not delved into urbackups concepts related to image backups since I don’t routinely use images, although I’d guess that the concept is similar.
Note: This is also a discussion of HOW I UNDERSTAND URBACKUP CONCEPTS. I could be wrong! I don’t think I am, but keep in mind that I may not be stating everything perfectly.
I won’t go over the traditional concepts in detail because I think everybody probably already knows those. i.e., a traditional incremental backup will not work without all the other incrementals that came before it, back to the last full backup, which is required as well. To restore, you start with the latest full, then apply all the incrementals, in order, up to the latest incremental.
Say you did a full backup and it ended up containing 100 files. Then two files changed, and you did an incremental backup. The incremental would contain 2 files. To restore, you first restore the 100 files from the full. Then you restore from the incremental which will overwrite 2 of the files that were previously restored from the full. The end result being that you have restored the newest version of all 100 files (98 of them came from the full and 2 came from the incremental). This is the traditional concept. In this example the full contains 100 files, and the incremental contains 2 files.
But the above is not what urbackup does.
A full backup in urbackup’s concept means every file is copied to backup. Whether it was there before or not, whether it had changed or not. i.e., a “fresh” copy of the file is created in the backup. This is the same as the traditional concept of a full backup. So a urbackup full would contain 100 files. Same as traditional.
Now when urbackup does an incremental, it creates that containing the 2 changed files. BUT THEN IT LINKS THE 98 UNCHANGED FILES FROM THE FULL INTO THE INCREMENTAL. So the urbackup incremental will contain all 100 files. Two of those files will be newly written data, and 98 of them will be linked from the previous full backup. So with urbackup, BOTH the full AND the incremental will each contain 100 files. So you can delete the full and the incremental still hangs around WITH ALL 100 FILES. This is the difference between the traditional concept and the urbackup concept.
One might say, “Then why do we ever have to do a full backup?” And the answer is “You don’t!” You can just do nothing but incrementals. Including the very first backup you do. If you tell urbackup to do an incremental and there are no previous backups to compare against to see what files changed, then that first incremental IS a full backup.
So the next question one might ask is, “Then why does urbackup even give you the capability to do a full backup?” Well, there are some times when you might want to. Say you’ve been doing nothing but daily incrementals for 5 years. And say you’ve got a file that has never changed since the very first backup. So that file has been written to backup once, and now it has thousands of links to the same data from all the incrementals you have done since that first backup. Now say your hard disk storing the backups is getting old and corrupts a few bits on this file with multiple links to it. Now every single one of your backups contains this corrupted version of the file. Because all this time you only had one single copy of the file that was LINKED to many different incrementals. To mitigate this potential failure point you may want to do routine full backups every now and then. Because a full will FORCE this file - changed or not - to be re-written to backup, and it will be written to a different place on the backup disk than the original (potentially corrupted) copy of that same file is written to.
If you have perfect backup media that is guaranteed never to fail, ever, then there is no need to ever do a full backup in urbackup. However, since perfect backup media does not exist, you probably want to do a full backup once every couple of months, once a month, once a week or at whatever interval you are willing to risk your backups to - i.e., how long to you trust your backup disk to remain “perfect”? As your backup disk gets older and older you might want to protect yourself by doing more frequent full backups.
You are contradicting yourself.
No, you can NOT do an incremental without another backup to use as reference.
For fun, I even TRIED starting with an incremental on a new client using btrfs. It makes a full backup.
You HAVE TO create a reference, THEN you can do incrementals forever.
You can even remove the first full after an incremental because urbackup MOVES the files in the snapshot to the next snapshot in line when deleted.
An incremental BUILDS upon another backup, be it another incremental or a full, but you can NOT START with an incremental, that is a FULL. Be it that Urbackup does that FOR YOU so you don’t really notice, but it is STILL A FULL, not an incremental that is the absolute first backup.
This is exactly what I said. In the very lines from my post that you quoted.
You tell urbackup to do an incremental as the very first backup. The backup that is done will be a full. Regardless that you did not tell urbackup to do a full, you told it to do an incremental. It did a full anyway.
Maybe I could state it like this to clarify: “You never have to TELL urbackup to do a full backup” (because if it needs a full, it will do one anyway, even though you technically told it to do an incremental).
And where, pray tell, will those 98 links point if the source of said links no longer exists. Links do not contain data, they point to the location of the actual data.
If (and I’m not doubting it, but it’s unusual) urbackup does as @anon16079863 says - moving files from the initial full to “the most recent incremental taken after it” when an initial full file backup was taken, it is, in essence, keeping the initial full file backup (which is very, very unusual in the world of full followed by either incremental or differential backups). Urbackup essentially creates a “moving full backup baseline,” pushing the original full into the next available incremental, if it’s removed. It also sounds like it does the same thing if you were tl delete the “incremental that now functions as the full.” The files in that incremental that functions as the full would be pushed into the next incremental up the line.
Personally, even though this works, I don’t see the value in deviating from the classic practices in this way. It makes things more confusing, not less so, and that, in my opinion, is a very bad thing. It’s so much more clear to retain the “must have an initial full, known as full, and subsequent incrementals.” But I’m not the developer, so those are choices I don’t get to make.
You need to research the difference between hard links and symlinks. Symlinks (a.k.a. symbolic links) behave as you describe. Hard links do not. Urbackup uses hard links. Urbackup may use some symlinks for a few things - a symlink linking to the “most recent” backup would make sense. But for files that are part of a backup, hard links are used. Note that for some filesystem types - btrfs being one - urbackup uses snapshots and features of the file system in much the same way (conceptually) as it uses hard links in other file system types.
I used “simple language” for that statement. I didn’t want to expand to deep into the tecnicality of it. But yes, that statement is true, be it a snapshot that is relinked or hard/symlinks, it depends on the setup you created for urbackup.
The point is UrBackup is doing the heavy lifting in this, you don’t really have to care HOW (unless you are curious ofc)
What I get from that is “I don’t understand so it has to be a bad practice”.
You not understanding has VERY LITTLE to do with “best practice”. xD
And what you get from that is what you get from that.
Terminology surrounding backup and recovery and how it works is many decades old. There is a clear understanding of what full/baseline, incremental, and differential are SUPPOSED to mean in that arena.
I don’t give a flying rat’s ass how “superior” anyone thinks that UrBackup’s method (or any product’s method) is. When it confuses and conflates by muddling very well known and well understood terminology, it’s problematic.
And what’s been described here causes a real mess in the understanding of full/baseline versus incremental backups for most of the world that is doing them or has done them for a very, very long period of time. I think that’s a problem, you don’t.
It took you a while to finally get it so you are frustrated - that’s normal and I don’t mean to be disrespectful in any way - but you can’t just come here and talk in the name of most of the world
You are doing backups for many many years so I take you are aware of always incremental backup scheme. There is a nice post on Bareos site here. I wonder how it fits to your understanding of most of the backup world? What UrBackup does is not THAT much different, is it.
EDIT:
One more thing, related to this part:
It makes me less confident that you know always incremental scheme. Please spend 10 minutes on reading the Bareos link I provided, it describes (and illustrates) possible advantages very nicely - and then decide if it’s worth it for you or not.
I read that article and it’s informative. Note, and note well, that Bareos uses a specific type designation, “Always Incremental,” to describe what’s going on. It makes it very clear that “classic incremental” and “always incremental” for file backups are not one and the same thing.
There seems to be a misperception that I don’t understand what’s going on, nor why it’s advantageous, neither of which is true.
What I object to is that it is not, clearly, differentiated from “classic” incremental in UrBackup. It is clearly differentiated, both in documentation and syntax, in Bareos. That matters.
But with this I’m out, as that Bareos article makes clear, very clear, that the entire “always incremental” scheme is only appropriate when talking about file backups.
I originally entered this topic because someone asked about “how it would be different” were full system images being discussed. And you don’t ever do “always incremental” in that context. It’s still classic full followed by classic incremental, and if you were to delete the baseline full system image backup all incremental backups upon which it depends are no longer of any use.
This has always been an objection on using a specific and well-known terminology for a backup type that is not, in fact, that backup type. “Always incremental” actually involves mechanisms not characteristic of “classic incremental,” and that’s fine. But that “Always” makes it clear that something else is being done. Personally, I’d have called it something like “Roll-up Incremental,” but it really doesn’t matter what it’s called provided it’s clearly identified/differentiated from plain, vanilla incremental as it’s been understood for longer than I’ve been alive.
Thanks. Terms have meanings, and some have far more well-established meanings than others. It’s wise to use either a completely fresh one, or one with a distinct modifier, when you are not, in actuality, doing what the well-established term means.
Giving distinct actions distinct identifiers is not a trivial request. It promotes understanding and prevents unnecessary confusion.
I wish this were the case. But it can be just the opposite. Example: The “OK” sign I’ve used all my life. Thumb and index finger touching forming a circle, other fingers extended. Forever it meant “I’m all right.” “Everything is fine.” “I understand.” “I agree with you.” But now it means something about white supremacy (I think?), and we are all supposed to be aware of this, otherwise we’ll be harassed, probably beat up, and maybe even prosecuted for some hate crime. There are a bazillion terms whose meaning has evolved, or in the case of this “OK” sign, have been replaced by a totally unrelated meaning and all previous meanings are to be abandoned.
Technically, Urbackups concept of “incremental” is not far from the conventional concept. When doing incrementals, it does indeed backup and write files to the backup media that have changed since the last full/incremental. However, it additionally adds links to the previous incrementals and full. So that those are no longer required at restore time. And you don’t have the added complexity of having to keep all the associated previous backups. But if you want to handle a restore using the conventional method, you certainly can. You can restore the full, and then all the incrementals in sequence - the conventional way (assuming you saved them). And you will end up with a perfectly good restore. But you don’t have to do all that - you can just use Urbackups slight enhancement to the conventional incremental setup (adding the links) to save yourself some time and complexity. But it’s your choice either way.
You do realize that the example you lead with reinforces my point. It’s a classic example of a well-established symbol/term being co-opted, and the confusion arises when the co-opted usage begins to be in the cultural ascendancy. It’s very analogous to what I’m decrying about the use of naked “incremental” being used in a way that incremental had never traditionally been. It’s even worse when you’ve gotta handle “full plus incrementals” one way for system image backups and another for file backups (where the full can be jettisoned at will).
At this point I do have to withdraw, because I believe that everyone involved in this exchange clearly knows what the others are saying, but we are not and will never be in agreement regarding whether the re-use of existing terminiology for a novel (and it is still, relatively speaking, novel) backup concept is a good idea. I think the concept itself, expressed in an article previously cited as, “Always incremental,” is a phenomenal one for file backups and makes sense. It should be clearly differentiated from classic incremental, which UrBackup does not do well because it just calls it incremental, and incremental for file backups means something different than incremental for system image backups.
We are all in our own camps and, on the whole, are likely to stay there. It’s been an interesting exchange all around.
Yeah, this thread did go a bit out of line… @anon16079863
And yeah, the terminology is a bit sub-optimal. As an excuse I was young and stupid when I did that and copied it from backuppc.
If you have constructive hints on how this situation can be improved I’m all ears. E.g. I’m often thinking that it might be stable enough to do away with the “full” backups completely. So just have “File backup” and “Resumed file backup” everywhere.
