Hello,
after using urbackup for a while now I noticed some weird behaviour with the retention.
Lets say I have a client with lots of backups, like this:
Now as time passes by, the oldest one, number 9, will be deleted in the nightly job. But that leaves me with number 7 and 8, two incrementel fullbackups without a corresponding full.
Are those 2 incrementals still functional, so do they still keep the whole content?
Thanks.
UrBackup is designed to be very efficient and to make each backup look like a full backup when it comes to restoring files. The âbitsâ of the backed-up files are stored separately from the metadata, descriptive information about a file as shown in a directory listing. If one file is stored in two places, even on different machines, UrBackup will keep one copy of the file and use features built into the operating system to give you a view as if each backup had a full copy of the files in their proper locations.
Full backups here mean the Client program tells the Server program about every file on the client computer. Incremental means the Client checks the list it made from the last full backup and only tells the Server about files that have changed. In either case only files that donât already exist on the server computer are transferred in full. The changes to the metadata are all thatâs needed otherwise. Incremental backups tend to run faster so many admins set their systems up to run mostly incrementals and very few fulls.
tl;dr - For file restore purposes there is no difference inside UrBackup between âfullâ and âincrementalâ backups. Both contain the full file data as of the time of the backup.
For file restore purposes there is no difference inside UrBackup between âfullâ and âincrementalâ
backups. Both contain the full file data as of the time of the backup.
So what are the pros/cons of keeping 10 increments, or 100, or 1000⌠or 0?
If UrBackUp automatically deletes 50 of my 100 increment backups⌠nothing is lost?
General Notes
Many users of UrBackup choose to only do Incremental File backups as they are confident that nothing is being missed by the UrBackup Client. In my business I mostly see computers that are not working well, so I tend to be less trusting. I find that daily incremental backups and monthly full backups are sufficient for the offices I manage. Others may want to run quick incrementals several times a day to avoid losing work in progress, such as creative writing or large legal documents that may be harder to redo if lost. Full backups impose an additional burden on the Client and Server to examine and report on each file within the directories marked for backup, whether or not the Client thinks it was changed, so count them as more expensive in computing resources.
UrBackup treats the maximum number of full or incremental backups as an âup toâ limit. Given infinite storage, no more than that number will be maintained. Each backup has the overhead of the file structure and accounting info needed to create the appearance of a full backup. More frequent backups and more backup copies being kept will shift a larger percentage of your storage to this overhead instead of keeping actual file changes, which may get dropped as storage fills up and UrBackup deletes older copies to make room.
Interlude: Microsoft Windows doesnât make things easy.
Linux and UNIX-style operating systems have a reasonably well defined storage hierarchy of things that should be kept forever, things that are expected to be discarded after a process terminates, and things that are in-between, the equivalent of a whiteboard with current project status notes that can be re-created easily but are more convenient to keep close at hand. This lets the administrator assign priorities for backup items and only copy what needs to be retained.
Windows is that messy office with stuff in piles, where every flat surface has stacks of documents from two-year-old lunch receipts to policy manuals to birth certificates and family movies of the kidâs first steps. Backing up the Users directory gets all of that stuff, including the crumpled notes thrown at the overflowing trashcan. Yes, there is a Temp directory you can exclude, several in fact, but each application (looking at you, Google) likes to keep its own cache of precious stuff like pictures from the website you visited by accident six months ago. That all gets dumped into the hidden AppData directory within the Users folder where itâs hard to see. In my offices, every incremental backup I look at is mostly the garbage and extra working copies of files, stuff that changes frequently, with very little new âkeep foreverâ data. As much as possible I try to limit things to the Desktop and Documents folders, and let the monthly Image backup catch the rest.
Choosing a Backup Schedule
Type 1: Youâve got tons of storage in fast RAID pairs, a dedicated local UrBackup Server, a separate offsite UrBackup Server for the critical stuff, and really fast workstations that can do an incremental without the users even noticing. Go ahead and cast a wide net and do frequent backups with large retention counts, just in case Pat wants that draft from four months ago - itâs something to do with apples.
Type 2: You donât even have enough storage to get a Full Image backup of all the sets for disaster recovery. Set UrBackup for Incremental File backups of only carefully selected folders, and donât expect to keep things longer than a week. Your Disaster Recovery backup will be on a portable USB hard drive with a free copy of Macrium Reflect that you walk from set to set outside working hours as you have time. If youâre lucky you can get the cost of the drive reimbursed. Professional satisfaction will have to do for the rest.
Type 3: Your UrBackup server is also the main file and database server, and itâs also the only server in the office. You get a single space to install a hard drive, as long as you move a pair of SSDs into an unused floppy bay. You dream of the new 12 Terabyte models but settle for an eBay special of a 4TB drive that backblaze.com says was pretty reliable for them. (This is my world.) You do what you can to restrict backup selections so you avoid most of the daily garbage, and prioritize keeping at least two image backups for disaster recovery because Windows will find something wrong with at least one of them. (You also use Macrium Reflect on critical workstations because only one type of backup is no backup at all.) Let UrBackup manage whatâs left over and watch that single backup drive for early signs of trouble.
Hope this helps! --Don
Kinda hidden in that novella was the answer to your question. Think of each backup as a point-in-time copy of the files at that moment. When UrBackup manages storage by deleting older backups, it will also notice the files that are not needed for later backups (because of changes or deletion since then) and purge the no-longer-needed copies from storage. So yes, things can be lost, but not generally anything currently present on the sets being backed up.
Note also the special case of files that are present on several sets. These will only be deleted after all backups that refer to that version of the file have been deleted. If everyone has the 2021 Policy Manual but April is also keeping the 2019 version she worked on, UrBackup will have to keep both versions in storage, but only one copy of each. The 2020 Manual is no longer saved on any set, so as soon as the last old backup from 2020 is deleted, it will vanish as well.
tl;dr - For file restore purposes there is no difference inside UrBackup between âfullâ and âincrementalâ backups. Both contain the full file data as of the time of the backup
What if Iâm NOT doing a âfile restoreâ but rather a âfull system restoreâ?
How does that change your statement?
While it is possible you might get a reply, adding to a topic when itâs already 2 years after the most recent prior post doesnât give that high odds.
I still donât understand whatâs being said in this topic myself, as it violates what I know of both incremental and differential backups, neither of which should âworkâ at restore time unless there is a baseline full backup. Iâm not seeing anything near to the amount of data backed up, file-size wise, in my incremental backups such that it would ever be able to use one to do a full system image restore were there no full system image to start with then apply the changes in each incremental backup taken since.
And when I say restore, I mean a âbare metalâ restore on a system where the main drive thatâs being backed up has failed entirely.
Itâs worth having a read through a couple of the search results from: Backup Types - Full, Differential, and Incremental
Well, you start talking about image backups while all above is about FILE backups soooo, thereâs that.
Depending on what filesystem is being used incremental only backups are absolutely a thing where no full file backup is needed for fallback (after the absolute first one is done that then can be deleted when an incremental backup is made).
I do them and have been doing incremental only for a loooooooong time.
Youâre right, if you accept that premise. But thereâs nothing in the first post that indicates file versus system. Itâs the follow-ups that presume file backups and it continues from there.
But, youâre 100% correct if what weâre talking about is file backups (except, perhaps, if something is deleted straight out of the shoot after a full and any one of the incrementals from which it could be restored gets nuked).
I did make my context clear, and I stand by what I said when it comes to full system image backups. If you donât have the baseline full upon which the incrementals are applied, then the incrementals are useless.
In any backup scenario, file or full system image, I always follow a protocol where thereâs an underlying full backup with however many incrementals until the next full gets taken. Then I may, or may not, depending on the circumstance, nuke the prior sequence of full and all associated incrementals.
Also note that the ârevival postâ asked specifically about full system image backups rather than file backups. You want a baseline full in that case, always.
When you make the ABOLUTE FIRST INCREMENTAL, yes, and nobody is claiming the opposite, not sure why you are arguing.
AFTER that it depends on stuff if it matters whether you can delete the full or not.
I once even tried to start with an incremental just to see how urbackup reacted, and it just made a full.
About image backups.
That should be telling you something . . .
Since the question that got me into this thread was directly premised on how this process is different for full system image backups plus follow-up incrementals I stand by what I said. You cannot just toss away the full system image baseline and deal with only incrementals in that case.
Iâve been dealing with system backup protocols for decades now, and for system images, you either always take a full and keep it along with all its incrementals to do âbare metal recoveryâ or you just consistently take full system image backups at some reasonable interval so you can recover from a full image backup alone. Itâs also faster to recover that way, though the backup itself is a lot slower. The reason incrementals came about in the context of full system image backing up is so speed of backup could be radically increased, but recovery time then suffers. There is no free lunch.
I think to clarify the question, Jill is asking for image backups, will a full image backup be deleted by UrBackup cleanup if there are newer incremental image backups that still reference back to it?
âYou can select on which backup incremental image backups should be based. Either on the last full or incremental image backup or on the last full backup (sometimes called differential). Keep in mind that image backups can only be deleted if all the image backups they are based on are also deleted.â
https://www.urbackup.org/administration_manual.html#x1-720008.5.3
This is because there is the traditional concept of full, incremental and differential backups - what you are familiar with - and then there is the urbackup concept of full and incremental backups.
THESE TWO CONCEPTS ARE DIFFERENT EVEN THOUGH THEY SHARE THE SAME NAMES!
The discussion below relates to file backups, not image backups. I have not delved into urbackups concepts related to image backups since I donât routinely use images, although Iâd guess that the concept is similar.
Note: This is also a discussion of HOW I UNDERSTAND URBACKUP CONCEPTS. I could be wrong! I donât think I am, but keep in mind that I may not be stating everything perfectly.
I wonât go over the traditional concepts in detail because I think everybody probably already knows those. i.e., a traditional incremental backup will not work without all the other incrementals that came before it, back to the last full backup, which is required as well. To restore, you start with the latest full, then apply all the incrementals, in order, up to the latest incremental.
Say you did a full backup and it ended up containing 100 files. Then two files changed, and you did an incremental backup. The incremental would contain 2 files. To restore, you first restore the 100 files from the full. Then you restore from the incremental which will overwrite 2 of the files that were previously restored from the full. The end result being that you have restored the newest version of all 100 files (98 of them came from the full and 2 came from the incremental). This is the traditional concept. In this example the full contains 100 files, and the incremental contains 2 files.
But the above is not what urbackup does.
A full backup in urbackupâs concept means every file is copied to backup. Whether it was there before or not, whether it had changed or not. i.e., a âfreshâ copy of the file is created in the backup. This is the same as the traditional concept of a full backup. So a urbackup full would contain 100 files. Same as traditional.
Now when urbackup does an incremental, it creates that containing the 2 changed files. BUT THEN IT LINKS THE 98 UNCHANGED FILES FROM THE FULL INTO THE INCREMENTAL. So the urbackup incremental will contain all 100 files. Two of those files will be newly written data, and 98 of them will be linked from the previous full backup. So with urbackup, BOTH the full AND the incremental will each contain 100 files. So you can delete the full and the incremental still hangs around WITH ALL 100 FILES. This is the difference between the traditional concept and the urbackup concept.
One might say, âThen why do we ever have to do a full backup?â And the answer is âYou donât!â You can just do nothing but incrementals. Including the very first backup you do. If you tell urbackup to do an incremental and there are no previous backups to compare against to see what files changed, then that first incremental IS a full backup.
So the next question one might ask is, âThen why does urbackup even give you the capability to do a full backup?â Well, there are some times when you might want to. Say youâve been doing nothing but daily incrementals for 5 years. And say youâve got a file that has never changed since the very first backup. So that file has been written to backup once, and now it has thousands of links to the same data from all the incrementals you have done since that first backup. Now say your hard disk storing the backups is getting old and corrupts a few bits on this file with multiple links to it. Now every single one of your backups contains this corrupted version of the file. Because all this time you only had one single copy of the file that was LINKED to many different incrementals. To mitigate this potential failure point you may want to do routine full backups every now and then. Because a full will FORCE this file - changed or not - to be re-written to backup, and it will be written to a different place on the backup disk than the original (potentially corrupted) copy of that same file is written to.
If you have perfect backup media that is guaranteed never to fail, ever, then there is no need to ever do a full backup in urbackup. However, since perfect backup media does not exist, you probably want to do a full backup once every couple of months, once a month, once a week or at whatever interval you are willing to risk your backups to - i.e., how long to you trust your backup disk to remain âperfectâ? As your backup disk gets older and older you might want to protect yourself by doing more frequent full backups.
You are contradicting yourself.
No, you can NOT do an incremental without another backup to use as reference.
For fun, I even TRIED starting with an incremental on a new client using btrfs. It makes a full backup.
You HAVE TO create a reference, THEN you can do incrementals forever.
You can even remove the first full after an incremental because urbackup MOVES the files in the snapshot to the next snapshot in line when deleted.
An incremental BUILDS upon another backup, be it another incremental or a full, but you can NOT START with an incremental, that is a FULL. Be it that Urbackup does that FOR YOU so you donât really notice, but it is STILL A FULL, not an incremental that is the absolute first backup.
This is exactly what I said. In the very lines from my post that you quoted.
You tell urbackup to do an incremental as the very first backup. The backup that is done will be a full. Regardless that you did not tell urbackup to do a full, you told it to do an incremental. It did a full anyway.
Maybe I could state it like this to clarify: âYou never have to TELL urbackup to do a full backupâ (because if it needs a full, it will do one anyway, even though you technically told it to do an incremental).
Thatâs the concept I was trying to convey.
And where, pray tell, will those 98 links point if the source of said links no longer exists. Links do not contain data, they point to the location of the actual data.
If (and Iâm not doubting it, but itâs unusual) urbackup does as @bedna says - moving files from the initial full to âthe most recent incremental taken after itâ when an initial full file backup was taken, it is, in essence, keeping the initial full file backup (which is very, very unusual in the world of full followed by either incremental or differential backups). Urbackup essentially creates a âmoving full backup baseline,â pushing the original full into the next available incremental, if itâs removed. It also sounds like it does the same thing if you were tl delete the âincremental that now functions as the full.â The files in that incremental that functions as the full would be pushed into the next incremental up the line.
Personally, even though this works, I donât see the value in deviating from the classic practices in this way. It makes things more confusing, not less so, and that, in my opinion, is a very bad thing. Itâs so much more clear to retain the âmust have an initial full, known as full, and subsequent incrementals.â But Iâm not the developer, so those are choices I donât get to make.
You need to research the difference between hard links and symlinks. Symlinks (a.k.a. symbolic links) behave as you describe. Hard links do not. Urbackup uses hard links. Urbackup may use some symlinks for a few things - a symlink linking to the âmost recentâ backup would make sense. But for files that are part of a backup, hard links are used. Note that for some filesystem types - btrfs being one - urbackup uses snapshots and features of the file system in much the same way (conceptually) as it uses hard links in other file system types.
My bad, I misunderstood your statements.
I used âsimple languageâ for that statement. I didnât want to expand to deep into the tecnicality of it. But yes, that statement is true, be it a snapshot that is relinked or hard/symlinks, it depends on the setup you created for urbackup.
The point is UrBackup is doing the heavy lifting in this, you donât really have to care HOW (unless you are curious ofc)
What I get from that is âI donât understand so it has to be a bad practiceâ.
You not understanding has VERY LITTLE to do with âbest practiceâ. xD
And what you get from that is what you get from that.
Terminology surrounding backup and recovery and how it works is many decades old. There is a clear understanding of what full/baseline, incremental, and differential are SUPPOSED to mean in that arena.
I donât give a flying ratâs ass how âsuperiorâ anyone thinks that UrBackupâs method (or any productâs method) is. When it confuses and conflates by muddling very well known and well understood terminology, itâs problematic.
And whatâs been described here causes a real mess in the understanding of full/baseline versus incremental backups for most of the world that is doing them or has done them for a very, very long period of time. I think thatâs a problem, you donât.