Hello,
after using urbackup for a while now I noticed some weird behaviour with the retention.
Lets say I have a client with lots of backups, like this:
Now as time passes by, the oldest one, number 9, will be deleted in the nightly job. But that leaves me with number 7 and 8, two incrementel fullbackups without a corresponding full.
Are those 2 incrementals still functional, so do they still keep the whole content?
Thanks.
UrBackup is designed to be very efficient and to make each backup look like a full backup when it comes to restoring files. The “bits” of the backed-up files are stored separately from the metadata, descriptive information about a file as shown in a directory listing. If one file is stored in two places, even on different machines, UrBackup will keep one copy of the file and use features built into the operating system to give you a view as if each backup had a full copy of the files in their proper locations.
Full backups here mean the Client program tells the Server program about every file on the client computer. Incremental means the Client checks the list it made from the last full backup and only tells the Server about files that have changed. In either case only files that don’t already exist on the server computer are transferred in full. The changes to the metadata are all that’s needed otherwise. Incremental backups tend to run faster so many admins set their systems up to run mostly incrementals and very few fulls.
tl;dr - For file restore purposes there is no difference inside UrBackup between ‘full’ and ‘incremental’ backups. Both contain the full file data as of the time of the backup.
For file restore purposes there is no difference inside UrBackup between ‘full’ and ‘incremental’
backups. Both contain the full file data as of the time of the backup.
So what are the pros/cons of keeping 10 increments, or 100, or 1000… or 0?
If UrBackUp automatically deletes 50 of my 100 increment backups… nothing is lost?
General Notes
Many users of UrBackup choose to only do Incremental File backups as they are confident that nothing is being missed by the UrBackup Client. In my business I mostly see computers that are not working well, so I tend to be less trusting. I find that daily incremental backups and monthly full backups are sufficient for the offices I manage. Others may want to run quick incrementals several times a day to avoid losing work in progress, such as creative writing or large legal documents that may be harder to redo if lost. Full backups impose an additional burden on the Client and Server to examine and report on each file within the directories marked for backup, whether or not the Client thinks it was changed, so count them as more expensive in computing resources.
UrBackup treats the maximum number of full or incremental backups as an “up to” limit. Given infinite storage, no more than that number will be maintained. Each backup has the overhead of the file structure and accounting info needed to create the appearance of a full backup. More frequent backups and more backup copies being kept will shift a larger percentage of your storage to this overhead instead of keeping actual file changes, which may get dropped as storage fills up and UrBackup deletes older copies to make room.
Interlude: Microsoft Windows doesn’t make things easy.
Linux and UNIX-style operating systems have a reasonably well defined storage hierarchy of things that should be kept forever, things that are expected to be discarded after a process terminates, and things that are in-between, the equivalent of a whiteboard with current project status notes that can be re-created easily but are more convenient to keep close at hand. This lets the administrator assign priorities for backup items and only copy what needs to be retained.
Windows is that messy office with stuff in piles, where every flat surface has stacks of documents from two-year-old lunch receipts to policy manuals to birth certificates and family movies of the kid’s first steps. Backing up the Users directory gets all of that stuff, including the crumpled notes thrown at the overflowing trashcan. Yes, there is a Temp directory you can exclude, several in fact, but each application (looking at you, Google) likes to keep its own cache of precious stuff like pictures from the website you visited by accident six months ago. That all gets dumped into the hidden AppData directory within the Users folder where it’s hard to see. In my offices, every incremental backup I look at is mostly the garbage and extra working copies of files, stuff that changes frequently, with very little new “keep forever” data. As much as possible I try to limit things to the Desktop and Documents folders, and let the monthly Image backup catch the rest.
Choosing a Backup Schedule
Type 1: You’ve got tons of storage in fast RAID pairs, a dedicated local UrBackup Server, a separate offsite UrBackup Server for the critical stuff, and really fast workstations that can do an incremental without the users even noticing. Go ahead and cast a wide net and do frequent backups with large retention counts, just in case Pat wants that draft from four months ago - it’s something to do with apples.
Type 2: You don’t even have enough storage to get a Full Image backup of all the sets for disaster recovery. Set UrBackup for Incremental File backups of only carefully selected folders, and don’t expect to keep things longer than a week. Your Disaster Recovery backup will be on a portable USB hard drive with a free copy of Macrium Reflect that you walk from set to set outside working hours as you have time. If you’re lucky you can get the cost of the drive reimbursed. Professional satisfaction will have to do for the rest.
Type 3: Your UrBackup server is also the main file and database server, and it’s also the only server in the office. You get a single space to install a hard drive, as long as you move a pair of SSDs into an unused floppy bay. You dream of the new 12 Terabyte models but settle for an eBay special of a 4TB drive that backblaze.com says was pretty reliable for them. (This is my world.) You do what you can to restrict backup selections so you avoid most of the daily garbage, and prioritize keeping at least two image backups for disaster recovery because Windows will find something wrong with at least one of them. (You also use Macrium Reflect on critical workstations because only one type of backup is no backup at all.) Let UrBackup manage what’s left over and watch that single backup drive for early signs of trouble.
Hope this helps! --Don
Kinda hidden in that novella was the answer to your question. Think of each backup as a point-in-time copy of the files at that moment. When UrBackup manages storage by deleting older backups, it will also notice the files that are not needed for later backups (because of changes or deletion since then) and purge the no-longer-needed copies from storage. So yes, things can be lost, but not generally anything currently present on the sets being backed up.
Note also the special case of files that are present on several sets. These will only be deleted after all backups that refer to that version of the file have been deleted. If everyone has the 2021 Policy Manual but April is also keeping the 2019 version she worked on, UrBackup will have to keep both versions in storage, but only one copy of each. The 2020 Manual is no longer saved on any set, so as soon as the last old backup from 2020 is deleted, it will vanish as well.
tl;dr - For file restore purposes there is no difference inside UrBackup between ‘full’ and ‘incremental’ backups. Both contain the full file data as of the time of the backup
What if I’m NOT doing a “file restore” but rather a “full system restore”?
How does that change your statement?
While it is possible you might get a reply, adding to a topic when it’s already 2 years after the most recent prior post doesn’t give that high odds.
I still don’t understand what’s being said in this topic myself, as it violates what I know of both incremental and differential backups, neither of which should “work” at restore time unless there is a baseline full backup. I’m not seeing anything near to the amount of data backed up, file-size wise, in my incremental backups such that it would ever be able to use one to do a full system image restore were there no full system image to start with then apply the changes in each incremental backup taken since.
And when I say restore, I mean a “bare metal” restore on a system where the main drive that’s being backed up has failed entirely.
It’s worth having a read through a couple of the search results from: Backup Types - Full, Differential, and Incremental
Well, you start talking about image backups while all above is about FILE backups soooo, there’s that.
Depending on what filesystem is being used incremental only backups are absolutely a thing where no full file backup is needed for fallback (after the absolute first one is done that then can be deleted when an incremental backup is made).
I do them and have been doing incremental only for a loooooooong time.
You’re right, if you accept that premise. But there’s nothing in the first post that indicates file versus system. It’s the follow-ups that presume file backups and it continues from there.
But, you’re 100% correct if what we’re talking about is file backups (except, perhaps, if something is deleted straight out of the shoot after a full and any one of the incrementals from which it could be restored gets nuked).
I did make my context clear, and I stand by what I said when it comes to full system image backups. If you don’t have the baseline full upon which the incrementals are applied, then the incrementals are useless.
In any backup scenario, file or full system image, I always follow a protocol where there’s an underlying full backup with however many incrementals until the next full gets taken. Then I may, or may not, depending on the circumstance, nuke the prior sequence of full and all associated incrementals.
Also note that the “revival post” asked specifically about full system image backups rather than file backups. You want a baseline full in that case, always.
When you make the ABOLUTE FIRST INCREMENTAL, yes, and nobody is claiming the opposite, not sure why you are arguing.
AFTER that it depends on stuff if it matters whether you can delete the full or not.
I once even tried to start with an incremental just to see how urbackup reacted, and it just made a full.
About image backups.
That should be telling you something . . .
Since the question that got me into this thread was directly premised on how this process is different for full system image backups plus follow-up incrementals I stand by what I said. You cannot just toss away the full system image baseline and deal with only incrementals in that case.
I’ve been dealing with system backup protocols for decades now, and for system images, you either always take a full and keep it along with all its incrementals to do “bare metal recovery” or you just consistently take full system image backups at some reasonable interval so you can recover from a full image backup alone. It’s also faster to recover that way, though the backup itself is a lot slower. The reason incrementals came about in the context of full system image backing up is so speed of backup could be radically increased, but recovery time then suffers. There is no free lunch.
I think to clarify the question, Jill is asking for image backups, will a full image backup be deleted by UrBackup cleanup if there are newer incremental image backups that still reference back to it?
“You can select on which backup incremental image backups should be based. Either on the last full or incremental image backup or on the last full backup (sometimes called differential). Keep in mind that image backups can only be deleted if all the image backups they are based on are also deleted.”
https://www.urbackup.org/administration_manual.html#x1-720008.5.3
This is because there is the traditional concept of full, incremental and differential backups - what you are familiar with - and then there is the urbackup concept of full and incremental backups.
THESE TWO CONCEPTS ARE DIFFERENT EVEN THOUGH THEY SHARE THE SAME NAMES!
The discussion below relates to file backups, not image backups. I have not delved into urbackups concepts related to image backups since I don’t routinely use images, although I’d guess that the concept is similar.
Note: This is also a discussion of HOW I UNDERSTAND URBACKUP CONCEPTS. I could be wrong! I don’t think I am, but keep in mind that I may not be stating everything perfectly.
I won’t go over the traditional concepts in detail because I think everybody probably already knows those. i.e., a traditional incremental backup will not work without all the other incrementals that came before it, back to the last full backup, which is required as well. To restore, you start with the latest full, then apply all the incrementals, in order, up to the latest incremental.
Say you did a full backup and it ended up containing 100 files. Then two files changed, and you did an incremental backup. The incremental would contain 2 files. To restore, you first restore the 100 files from the full. Then you restore from the incremental which will overwrite 2 of the files that were previously restored from the full. The end result being that you have restored the newest version of all 100 files (98 of them came from the full and 2 came from the incremental). This is the traditional concept. In this example the full contains 100 files, and the incremental contains 2 files.
But the above is not what urbackup does.
A full backup in urbackup’s concept means every file is copied to backup. Whether it was there before or not, whether it had changed or not. i.e., a “fresh” copy of the file is created in the backup. This is the same as the traditional concept of a full backup. So a urbackup full would contain 100 files. Same as traditional.
Now when urbackup does an incremental, it creates that containing the 2 changed files. BUT THEN IT LINKS THE 98 UNCHANGED FILES FROM THE FULL INTO THE INCREMENTAL. So the urbackup incremental will contain all 100 files. Two of those files will be newly written data, and 98 of them will be linked from the previous full backup. So with urbackup, BOTH the full AND the incremental will each contain 100 files. So you can delete the full and the incremental still hangs around WITH ALL 100 FILES. This is the difference between the traditional concept and the urbackup concept.
One might say, “Then why do we ever have to do a full backup?” And the answer is “You don’t!” You can just do nothing but incrementals. Including the very first backup you do. If you tell urbackup to do an incremental and there are no previous backups to compare against to see what files changed, then that first incremental IS a full backup.
So the next question one might ask is, “Then why does urbackup even give you the capability to do a full backup?” Well, there are some times when you might want to. Say you’ve been doing nothing but daily incrementals for 5 years. And say you’ve got a file that has never changed since the very first backup. So that file has been written to backup once, and now it has thousands of links to the same data from all the incrementals you have done since that first backup. Now say your hard disk storing the backups is getting old and corrupts a few bits on this file with multiple links to it. Now every single one of your backups contains this corrupted version of the file. Because all this time you only had one single copy of the file that was LINKED to many different incrementals. To mitigate this potential failure point you may want to do routine full backups every now and then. Because a full will FORCE this file - changed or not - to be re-written to backup, and it will be written to a different place on the backup disk than the original (potentially corrupted) copy of that same file is written to.
If you have perfect backup media that is guaranteed never to fail, ever, then there is no need to ever do a full backup in urbackup. However, since perfect backup media does not exist, you probably want to do a full backup once every couple of months, once a month, once a week or at whatever interval you are willing to risk your backups to - i.e., how long to you trust your backup disk to remain “perfect”? As your backup disk gets older and older you might want to protect yourself by doing more frequent full backups.
You are contradicting yourself.
No, you can NOT do an incremental without another backup to use as reference.
For fun, I even TRIED starting with an incremental on a new client using btrfs. It makes a full backup.
You HAVE TO create a reference, THEN you can do incrementals forever.
You can even remove the first full after an incremental because urbackup MOVES the files in the snapshot to the next snapshot in line when deleted.
An incremental BUILDS upon another backup, be it another incremental or a full, but you can NOT START with an incremental, that is a FULL. Be it that Urbackup does that FOR YOU so you don’t really notice, but it is STILL A FULL, not an incremental that is the absolute first backup.
This is exactly what I said. In the very lines from my post that you quoted.
You tell urbackup to do an incremental as the very first backup. The backup that is done will be a full. Regardless that you did not tell urbackup to do a full, you told it to do an incremental. It did a full anyway.
Maybe I could state it like this to clarify: “You never have to TELL urbackup to do a full backup” (because if it needs a full, it will do one anyway, even though you technically told it to do an incremental).
That’s the concept I was trying to convey.
And where, pray tell, will those 98 links point if the source of said links no longer exists. Links do not contain data, they point to the location of the actual data.
If (and I’m not doubting it, but it’s unusual) urbackup does as @anon16079863 says - moving files from the initial full to “the most recent incremental taken after it” when an initial full file backup was taken, it is, in essence, keeping the initial full file backup (which is very, very unusual in the world of full followed by either incremental or differential backups). Urbackup essentially creates a “moving full backup baseline,” pushing the original full into the next available incremental, if it’s removed. It also sounds like it does the same thing if you were tl delete the “incremental that now functions as the full.” The files in that incremental that functions as the full would be pushed into the next incremental up the line.
Personally, even though this works, I don’t see the value in deviating from the classic practices in this way. It makes things more confusing, not less so, and that, in my opinion, is a very bad thing. It’s so much more clear to retain the “must have an initial full, known as full, and subsequent incrementals.” But I’m not the developer, so those are choices I don’t get to make.
You need to research the difference between hard links and symlinks. Symlinks (a.k.a. symbolic links) behave as you describe. Hard links do not. Urbackup uses hard links. Urbackup may use some symlinks for a few things - a symlink linking to the “most recent” backup would make sense. But for files that are part of a backup, hard links are used. Note that for some filesystem types - btrfs being one - urbackup uses snapshots and features of the file system in much the same way (conceptually) as it uses hard links in other file system types.
My bad, I misunderstood your statements.
I used “simple language” for that statement. I didn’t want to expand to deep into the tecnicality of it. But yes, that statement is true, be it a snapshot that is relinked or hard/symlinks, it depends on the setup you created for urbackup.
The point is UrBackup is doing the heavy lifting in this, you don’t really have to care HOW (unless you are curious ofc)
What I get from that is “I don’t understand so it has to be a bad practice”.
You not understanding has VERY LITTLE to do with “best practice”. xD
And what you get from that is what you get from that.
Terminology surrounding backup and recovery and how it works is many decades old. There is a clear understanding of what full/baseline, incremental, and differential are SUPPOSED to mean in that arena.
I don’t give a flying rat’s ass how “superior” anyone thinks that UrBackup’s method (or any product’s method) is. When it confuses and conflates by muddling very well known and well understood terminology, it’s problematic.
And what’s been described here causes a real mess in the understanding of full/baseline versus incremental backups for most of the world that is doing them or has done them for a very, very long period of time. I think that’s a problem, you don’t.