Special treatment of .pst files (not .ost files)

Hello Uroni,

I think we need to find a clever way to backup outlook .pst files.

Let me explain.
We are in a company with 100+ people. Each user archives his mail in so called .pst files (outlook archive files). Some user have multiple .pst files. Some of them are quiet big. For example 4GB+

Unfortunately outlook always “touches” the file (change the file date) as soon as you open outlook. :frowning:
Result: Every day several GB are backup-up for nothing, because the content of the .pst didn’t change (files size is equal)

We need to find a clever way to backup these files.

I have two suggestions:
-.- Only store the differences as a “patch” (binary file diff) see this discussion Mozy backup and a possible utility here
-.- Check on file size only in case of .pst files. (needs investigation if this is valid)
-.- …

What’s your idea about this? Maybe it is out of scope?

When we nagged this issue, we go live :smile:

MartijnS

If you set the local transfer mode of incremental backups to “block differences - hashed” it should only transfer changed blocks in the file. Internally UrBackup uses a binary patch for that.

Saving only changed blocks on the server is only possible using btrfs.

If the file isn’t changed it gets hard-linked (saved only once) in any case.

Thanks that sounds great.
I will test it.

Hello Martin,

I changed the local transfer mode of incremental backups to “block differences - hashed”
And it works as expected. But unfortunately there is still a huge amount of data is transfered.

For example user 1:
He has two pst files: 5,1GB and 2,4GB
When he just opens outlook and close it again (no emails are moved to one of the .pst fils) the diff is 2,4GB!?!

Why are these files changed by just open and close them? Stupid Micro$oft…

I noticed that the file size doesn’t change when moving files to one of the archives. So I think I am stuck here.

One possible solution (to my problem!) could be to make the Excluded files (with wildcards) and Included files (with wildcards) “backup type (incremental of full) dependent”. This would make it possible for me to exclude the .pst files from the incremental file backups.

Anyhow it would be great to define these Include and exclude files path dependent. But this is a nice to have :slight_smile:

What do you think?

With kind regards
Martijn

I’ve found a bug, that causes it to transfer too much. It will be fixed in 1.4 RC1. That is probably causing it to transfer that much. Sorry.

Different excludes/includes will have to wait till different backup groups are implemented (those different groups can then have different exclude/includes).

Would this affect exchange stores too? I imagine that’s pretty similar?

Yes. RC1 is released now and fixes this. It makes the situation worse if you use 1.3 client with a 1.4 server and vise versa.

Hello

I just installed version 1.4 (client and server) but the result is not good.
The size of the incremental backup is now equal to the full. 7,2 GB :frowning:

Local transfermode for incremental backups was set to “block differences - hashed”

Do I need to do something more?

It corrects the checksums on the first full/incremental file backups. So it has to transfer more the first time. Can you try another one, to see if it works then?

This is what I did:

  • Delete all the previous backups (empty the btrfs disk)
  • Make a full file backup (several files and two big .pst files) => Used storage 12,33 GB (27 minutes)
  • Make an incremental file backup => Used storage 7,21GB (actualy the size of both pst files) (6 minutes)
  • Make a second incremental file backup Used storage 7,21GB (again the size of both pst files) (6 minutes)

Do I need to do some more tests?

Can you have a look at the logs, and see how much it transferred? The transfer mode does not affect how much storage is uses.
As for that the sizes shown on the web interface: They are only accurate to the whole file level, so that could be correct.

If you want to find out how much storage was really used, you need to use btrfs tools. See e.g. this blog post: http://dustymabe.com/2013/09/22/btrfs-how-big-are-my-snapshots/

As you can see, even then it is difficult, as the different snapshots can share data. If you use block level deduplication it’s even more confusing.

I have checked the log file and this seems to be okay.
In case of an incremental backup only 8.6 MB is transfered! Perfect. It takes approx 6 minutes to “build” the files.

But when I try to check the size of each snapshot I get this result:

sudo btrfs qgroup show /mnt/urbackup/ | /usr/local/bin/convert
qgroupid 0M 0M
-------- 0M 0M
0/5 42362M 42362M
0/339 0M 0M
0/340 0M 0M

After another incremental backup I get this
sudo btrfs qgroup show /mnt/urbackup/ | /usr/local/bin/convert
qgroupid 0M 0M
-------- 0M 0M
0/5 49748M 49748M
0/339 0M 0M
0/340 0M 0M

Another 7.2 GB is used
To me it seems that snapshots are not created properly!

When I run urbackup_snapshot_helper test
output is this:

sudo ./urbackup_snapshot_helper test
Create subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
Create a snapshot of ‘/mnt/urbackup/testA54hj5luZtlorr494/A’ in ‘/mnt/urbackup/testA54hj5luZtlorr494/B’
Delete subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
Delete subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/B’
TEST OK

Do you have some suggestions?

The test probably failed at startup time (it retrys so it must have failed a few times). The alternative being that it failed to create snapshots later on.

You can confirm UrBackup is in the btrfs mode and creating snapshots by running it in debug log mode.

If you run it via
start_urbackup_server --loglevel debug --no_daemon
you can even see the output of the test script.

I think I found the issue but I am unable to solve it.

when I start the server with this command
sudo start_urbackup_server --loglevel debug --no_daemon
the output is this:
Create subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
ERROR: cannot create subvolume - Permission denied
chown: kan geen toegang krijgen tot ‘/mnt/urbackup/testA54hj5luZtlorr494/A’: Bestand of map bestaat niet
TEST FAILED: Creating test subvolume failed
2014-09-04 09:05:39: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.

Permissions are 777 for the /mnt and /mnt/urbackup directory
Owner of /mnt = root:root
Owner of /mnt/urbackup is urbackup:urbackup

when I run
sudo ./urbackup_snapshot_helper test
output is:
Create subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
Create a snapshot of ‘/mnt/urbackup/testA54hj5luZtlorr494/A’ in ‘/mnt/urbackup/testA54hj5luZtlorr494/B’
Delete subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
Delete subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/B’
TEST OK

I am running urbackup on a Centos7 server.

The shell in centos seems to drop priviledges in system calls. Can you try this version where I have replaced the system calls? http://buildserver.urbackup.org/urbackup_unstable/urbackup-server-1.4.1.tar.gz
Seems to be better security vise anyways.

Sorry but the result is the same

Create subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
ERROR: cannot create subvolume - Permission denied
chown: kan geen toegang krijgen tot ‘/mnt/urbackup/testA54hj5luZtlorr494/A’: Bestand of map bestaat niet
TEST FAILED: Creating test subvolume failed
2014-09-05 11:08:41: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.

EDIT:
I tried it again and now I see this

2014-09-05 13:37:14: Testing if backup destination can handle subvolumes and snapshots…
Create subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
Create a snapshot of ‘/mnt/urbackup/testA54hj5luZtlorr494/A’ in ‘/mnt/urbackup/testA54hj5luZtlorr494/B’
Delete subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/A’
Delete subvolume ‘/mnt/urbackup/testA54hj5luZtlorr494/B’
TEST OK

I did a new incremental backup and the results at filesystem level are good now!

The statistcs tab still show the full 7,2GB but at filesystem level only 7MB is used! Great.

sudo btrfs qgroup show /mnt/urbackup/ | /usr/local/bin/convert
qgroupid 0M 0M
-------- 0M 0M
0/5 37510M 24861M
0/339 0M 0M
0/340 0M 0M
0/341 0M 0M
0/342 0M 0M
0/343 0M 0M
0/344 0M 0M
0/345 12685M 10M
0/346 12685M 0M
0/347 0M 0M
0/348 0M 0M
0/349 12685M 0M
0/350 12685M 7M
0/351 12685M 7M

I will continue testing.

You might want to disable the btrfs quota feature again afterwards. With my btrfs filesystem it had an extremely negative impact on performance.

thanks for the tip!
Question:
Any chance statistics are in line with the real diskspace used in case of btrfs snapshots?
Currently urbackup is deleting incremental backups due to “limited space”

If you don’t use the client quota it will only delete backups if it is really full (as per output of df).

I am running this for one week and everything seems works as expected!
Thank you very much.

Only on thing remains:
Statistics are not inline with the df output.

df:
Bestandssysteem Grootte Gebruikt Besch Geb% Aangekoppeld op
/dev/sdb 100G 28G 72G 28% /mnt/urbackup

urbackup statistics:
ab4lt-amf126 0 bytes 59.47 GB 59.47 GB
Showing 1 to 1 of 1 entriesFirstPrevious1NextLast