Warl archive: postgresql database backup failed with "Error saving metadata"

client
1

Hi all,

For the past few weeks, my wal archive backups (replication mode) of postgresql databases no longer work.
the error message is:

(error): Client pgisdata-new went offline.
(error): Error saving metadata. Could not open output file at "/media/vol1/urbackup/pgisdata-mshe.univ-fcomte.fr/221024-1932/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata"

this error seems unavoidable because urbackup_backup_scripts/postgresbase is a file and not a directory.

file -i /media/vol1/urbackup/pgisdata-mshe.univ-fcomte.fr/221024-1932/.hashes/urbackup_backup_scripts/postgresbase
: application/octet-stream; charset=binary

info:
urbackupclientctl --version: v2.5.20.0
urbackupsrv --version: v2.5.26.0
wal archive is configurated as explained in PosgreSQL: How to backup and restore PostgreSQL

In previous months this backup ran without problems, the .hashes/urbackup_backup_scripts/postgresbase/.dir_metadata file was created without any issues.

ls -ltr /media/vol1/urbackup/pgisdata/*/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata 
-rwx------ 1 urbackup urbackup 134 juin  18 03:53 /media/vol1/urbackup/pgisdata/220618-0004/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 juil.  2 03:43 /media/vol1/urbackup/pgisdata/220702-0003/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 août  13 10:34 /media/vol1/urbackup/pgisdata/220813-0000/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 août  27 15:54 /media/vol1/urbackup/pgisdata/220827-0000/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 sept.  3 04:09 /media/vol1/urbackup/pgisdata/220903-0002/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 sept. 10 16:10 /media/vol1/urbackup/pgisdata/current/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 sept. 10 16:10 /media/vol1/urbackup/pgisdata/220910-0004/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*

Have you ever encountered this error?

Thanks in advance,

Ernest.

2

We’re seeing the same issue and haven’t found a solution.
@Ernest_Chiarello have you found a solution in the meantime?

3

unfortunately no… I would have liked to declare this problem on the project’s github https://github.com/uroni/urbackup_backend, but it does not contain the “issues” section.
:anguished:

4

same problem here :frowning:
It seems to be the same as that one here which was reported in april?

5

The replication of the postgresql databases is not done, there is an error on this subject in the postgresql logs, /var/log/postgresql/postgresql-14-main.log in my case.

2022-11-22 10:31:45.164 CET [106175] postgres@[unknown] LOG: failed to send data to client: Relay broken (pipe)
2022-11-22 10:31:45.164 CET [106175] postgres@[unknown] INSTRUCTION: BASE_BACKUP LABEL 'pg_basebackup base backup' PROGRESS WAL NOWAIT TABLESPACE_MAP MANIFEST 'yes'
2022-11-22 10:31:45.165 CET [106175] postgres@[unknown] ERROR: base backup failed to send data, backup aborted
2022-11-22 10:31:45.165 CET [106175] postgres@[unknown] INSTRUCTION: BASE_BACKUP LABEL 'pg_basebackup base backup' PROGRESS WAL NOWAIT TABLESPACE_MAP MANIFEST 'yes'
2022-11-22 10:31:45.165 CET [106175] postgres@[unknown] FATAL: client connection lost

When I saw the new version of the Urbackup server, UrBackup 2.5.27, I thought that the problem was going to be solved… but such was not the case.

6

@Ernest_Chiarello we eventually got it working again.

  1. Downgrade the client to 2.4.11.
  2. Purge all backups in de backup-server

After we did that. The backups started working again (against newest server )

7

@thijs-42 you are right, thanks for the suggestion.
UrBackup Server v2.5.27.0 and UrBackup Client Controller v2.4.11.0 is the winning couple!
Disadvantage of this solution: updates from all other clients are blocked, but we can’t do better until the problem is solved.

8

Meanwhile I’m on Server 2.5.30. The problem still persists.
I looked into it again, as it seems the problem is the path.
Error saving metadata. Could not open output file at "/var/lib/urbackup/psqlserver/230419-1416/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata

I’ve taken a look at that directory. “.hashes/urbackup_backup_scripts/postgresbase” is a file. Therefore it can not contain the file .dir_metadata. But in “.hashes/urbackup_backup_scripts” there exists a file .dir_metadata. Probably it wan’t to reference that one?

Anyway, would be cool if there would be a bug tracker somewhere, to follow bugs, and get notified when they are fixed…

2 Likes