Full/incremental backups over the internet

#1

Hello,

I have a server that i use for a offshore backup server, that i was using Bareos and now i am migrating to Urbackup. All my clients are connected to it over the internet via a VPN. The clients have between 50Gb and 600Gb and one of then, 1.5Tb and the internet of the server is a 100Mbps Dsl home connection.

My question is, how can i deal with full file backup routines with this low speed and this among of data? I would like to have backup of the files for 3 months and most of the files don’t change a lot. Incremental file backups once a day are more then enough.
Should a configure something like:

Interval for incremental file backups: 24 hours
Interval for full file backups: 90
Maximal number of incremental file backups: 90?
Minimal number of incremental file backups:30?

Thanks a lot

Manny

#2

In the scope of server/clients configured to use “Internet mode,” the data is only transferred from the client to the server once. Only once. After your server has the data, it is smart enough to figure out which files have actually changed and truly need to be transmitted. This makes it sound like Internet mode causes incremental and full backups to be the same. However, that is not the case.
Incremental backups only contain files which have changed.
Full backups contain ALL files in the backup, HOWEVER, the server makes heavy use of what are called “hard links.” If the server determines that a given file, in a full backup, hasn’t changed on the client (when compared to what is backed up on the server) it will tell the client to not actually transmit that file. It (the server) will instead put a (I’m really simplifying) pointer to the file that was initially transmitted to the server. It won’t store or receive transmission of copy after copy of the same file at each full backup. This makes it so that space isn’t wasted over time. URbackup’s clever use of hard-links is one of the features which differentiates it from other backup solutions. : )
You will have to give it some number of days for the initial backup, there’s no getting around that. However, as long as your clients aren’t adding hundreds of GB before each 3 month interval, the on going backups will not be that much of a strain.
Anyway, hope this is helpful.

#3

You can seed your initial backup.
copy files to usb drive,
install client on server
plug usb drive to server,
backup the content of the usb drive, unplug it
backup the content of the real client
remove the client from the server

a suggestion for large dataset is to use virtual clients, look that in the doc, backup large unchanging data set less often.

1 Like
#4

Thanks a lot, that was exactly what i need to know, that the full backup will not recopy the data.

#5

Is that true? Is that noted in the manual?

So basically: If clients are connected through internet, even a full backup would transfer only needed data? I only found this in the manual:

If the backup is incremental only new and changed files are downloaded. If the backup is a full one all files are downloaded from the client.. No other notes, even in the internet section of the manual.

EDIT: Just found this:
What to do about fulls is up to you. If you are running Internet clients they are disabled by default. You can also set the minimum amount of full file backups to zero in this case.
=>

So if this is true, it shoud be noted at least clearly in the docs (sorry if it is, but I did not found it!). I would even go a step further and note that in the WebUI.

EDIT: Wait - are we talking about the internet_full_file_backups checkbox in the Internet tab (Allow full backups over internet)?

#6

Hmm. Tested a full backup on a client with recent full and incs: The full is actually an inc.

332.191 MB of files were already present on the server and did not need to be transferred → triggered by manual full.

Another full backup (internet) did that as well: 10.4626 GB of files were already present on the server and did not need to be transferred

A local client did a real full backup.

So:

  • This should be clearly mentioned in the manual
  • What is that internet_full_file_backups Checkbox then for, when an internet full is an inc? Or do I mix here something?

Maybe @uroni could explain that a bit :slight_smile:

EDIT: I disabled internet_full_file_backups on one of those clients and triggered a full backup: I cant see any difference.

#7

Ahh, the checkbox is for automatically running backups. If it is disabled and I run a manual full, the full is being executed. But the scheduler does not run full then. At least this is my observation.

#8

It’s a question of how you define “Create a full backup”

The old-fashioned method of “OK, copy everything then” which UrBackup doesn’t always use…
or the functionally equivalent “Read the drive content, get any data I don’t have, & present it just as if I’d transferred everything”. Which it quite often does for the sake of efficiency.

Restore either gets you the same result.

#9

Exact. And this is not that clear documented, isnt it?
Now I know (by testing) that internet clients full mean: only new stuff (inc). Is there any difference between internet full and inc at all then?

#10

You can turn this off by disabling Calculate file-hashes on the client in the Internet settings btw.

The main difference for incr vs. full file backup is that the full file backup will read all data from disk on the client. The server may still store it in a more efficient form (deduplication). Calculate file-hashes on the client just moves this deduplication so that data doesn’t need to be transferred.