Using postgresql_base backup regulary leads to hash errors

1

On one server I use urbackup to also backup the postgresql database regularly, by activating it in “postgresbase.conf”.
It works. Mostly.
A lot of times I get a hash missmatch error. See below.
The Size of the Database is around 50GB, everyday a few 100MB are added, and the backup is executed daily. I get the error message about 2-3 times per week, even though there are some weeks where I don’t get any message at all.
A few months ago I’ve lost all urbackup data because of an BTRFS error, and had to do a full backup again. It took about 20 tries, until it managed to copy the whole database without showing the error message.
Postgresql is running inside of a virtual machine with several other virtual machins running parallel. It uses ECC-Memory. And everything works perfectly stable (the “to backup server” and the “urbackup server”). Both servers run Linux. When doing only the filebackups without the postgresbase backup, no error ever appeared.
I can’t find a single error in any system logfiles, also smartdata of the HDDs look perfect.
So I don’t really believe there is a problem with the memory or harddisk and I also don’t believe in that many accidental bit flips.
Urbackup server: 2.4.12
Client: 2.4.9

that here is the error message:

ERROR: Client calculated hash of “/var/lib/urbackup/example.com/200226-2004/urbackup_backup_scripts/postgresbase/base/20352/21044.15” differs from server calculated hash. This may be caused by a bug or by random bit flips on the client or server hard disk. Failing backup. (Hash: sha512-nosparse, client hash: AspA3C6lZXUnY3NtbzNXabsHn8lExcKE63D+ZfcjYbz6gvIZOuEE9Mk85Qd+MviCW+RICrHKGR0M1m/jNnGPbw==, server hash: q/o/i0SAIv9yFQXVk92m06BcvAFrOeHHKpHpb4GaPZjiRjZd/7p2V0P1ReFFbiFCXFFzgcUYQaLTS4VoXCGKiA==)
ERROR: Client calculated hash of “/var/lib/urbackup/example.com/200226-2004/urbackup_backup_scripts/postgresbase/base/20352/21044.33” differs from server calculated hash. This may be caused by a bug or by random bit flips on the client or server hard disk. Failing backup. (Hash: sha512-nosparse, client hash: hJvC9cue0qRXpv8bpqHeRHybUatrDgIZunHa6X4oWhvKcxyd1lEqDAGBhMwIaWYx1oeLBO4YTaTh1HKuaV/9HA==, server hash: vemRjhOj64Z0W8/gAF16liBAS8ItG143+6mUYfzAaFd7fmpgNp9ZdrG+HLgfKpaCshDVtd29XPyEBs3Bu4/WDw==)
ERROR: FATAL: Backup failed because of disk problems (see previous messages)

Is it possible there is a bug somewhere?

2

Sorry you are having issues. To solve the problem the client (debug) log would be useful.
If possible, could you post it or send it?

This post describes how to change the client to debug logging, where it is stored and where to send it to if posting is not possible: Having problems with UrBackup? Please read before posting

Thanks!

3

Ok, I can see a problem in a previous fix. Can you compile the client or do you need a binary build?

4

wow. you’re way too fast. I’ve already started downloading the 140MB of logfiles with a 20kB/s connection :wink:
if you could tell me what dev dependencies (libraries,etc.) are necessary and give basic building instructions, I should be able to build it.

5

Ok, good. I guess you can apply this patch by hand to e.g. the 2.4.x tar.gz

6

ok. I’ve build it. let’s see if that fixes the error.
Am I right in the assumption that this patch is for the server? Or is it for the client, or both?

btw. The link to the patch was on github. But on github only release 2.4.9 of the urbackupserver is available, it took me a while to realize, where I could find 2.4.12…

7

It’s for the client. Source releases are linked at https://www.urbackup.org/download.html

8

Hi, you database in use when urbackup running there.

You have two choice:

  1. shutdown database when run backup
  2. make snapshot from that database disk

There is nothing wrong, that is normal error message!

9

@iseitten: The error message is not normal. pg_basebackup allows to take a consistent backup (snapshot) of a running database.

@uroni: since I’ve applied the above patch I’ve didn’t get the error message anymore. So most probably it’s working now.