Warl archive: postgresql database backup failed with "Error saving metadata"

[Ernest_Chiarello]

Hi all,

For the past few weeks, my wal archive backups (replication mode) of postgresql databases no longer work.
the error message is:

(error): Client pgisdata-new went offline.
(error): Error saving metadata. Could not open output file at "/media/vol1/urbackup/pgisdata-mshe.univ-fcomte.fr/221024-1932/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata"

this error seems unavoidable because urbackup_backup_scripts/postgresbase is a file and not a directory.

file -i /media/vol1/urbackup/pgisdata-mshe.univ-fcomte.fr/221024-1932/.hashes/urbackup_backup_scripts/postgresbase
: application/octet-stream; charset=binary

info:
urbackupclientctl --version: v2.5.20.0
urbackupsrv --version: v2.5.26.0
wal archive is configurated as explained in PosgreSQL: How to backup and restore PostgreSQL

In previous months this backup ran without problems, the .hashes/urbackup_backup_scripts/postgresbase/.dir_metadata file was created without any issues.

ls -ltr /media/vol1/urbackup/pgisdata/*/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata 
-rwx------ 1 urbackup urbackup 134 juin  18 03:53 /media/vol1/urbackup/pgisdata/220618-0004/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 juil.  2 03:43 /media/vol1/urbackup/pgisdata/220702-0003/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 août  13 10:34 /media/vol1/urbackup/pgisdata/220813-0000/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 août  27 15:54 /media/vol1/urbackup/pgisdata/220827-0000/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 sept.  3 04:09 /media/vol1/urbackup/pgisdata/220903-0002/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 sept. 10 16:10 /media/vol1/urbackup/pgisdata/current/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*
-rwx------ 1 urbackup urbackup 134 sept. 10 16:10 /media/vol1/urbackup/pgisdata/220910-0004/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata*

Have you ever encountered this error?

Thanks in advance,

Ernest.

[thijs-42]

We’re seeing the same issue and haven’t found a solution.
@Ernest_Chiarello have you found a solution in the meantime?

[Ernest_Chiarello]

unfortunately no… I would have liked to declare this problem on the project’s github https://github.com/uroni/urbackup_backend, but it does not contain the “issues” section.

[rmm]

same problem here
It seems to be the same as that one here which was reported in april?

Backup error using xtrabackup

Hello, Is there anything to fix this? The server is Windows and the client is CentOS 7. The database is MariaDB 10.2 Thanks 04/09/20 13:02 ERROR Error getting script output for command “mariadbxtrabackup” (err response) 04/09/20 13:02 DEBUG GT: Loaded file “mariadbxtrabackup” 04/09/20 13:02 ERROR Client staging-restore went offline. 04/09/20 13:02 DEBUG PT: Hashing file “mariadbxtrabackup” 04/09/20 13:02 DEBUG HT: Copying file: “C:\urbackup\staging-restore\200409-1256\urbackup_b…

[Ernest_Chiarello]

The replication of the postgresql databases is not done, there is an error on this subject in the postgresql logs, /var/log/postgresql/postgresql-14-main.log in my case.

2022-11-22 10:31:45.164 CET [106175] postgres@[unknown] LOG: failed to send data to client: Relay broken (pipe)
2022-11-22 10:31:45.164 CET [106175] postgres@[unknown] INSTRUCTION: BASE_BACKUP LABEL 'pg_basebackup base backup' PROGRESS WAL NOWAIT TABLESPACE_MAP MANIFEST 'yes'
2022-11-22 10:31:45.165 CET [106175] postgres@[unknown] ERROR: base backup failed to send data, backup aborted
2022-11-22 10:31:45.165 CET [106175] postgres@[unknown] INSTRUCTION: BASE_BACKUP LABEL 'pg_basebackup base backup' PROGRESS WAL NOWAIT TABLESPACE_MAP MANIFEST 'yes'
2022-11-22 10:31:45.165 CET [106175] postgres@[unknown] FATAL: client connection lost

When I saw the new version of the Urbackup server, UrBackup 2.5.27, I thought that the problem was going to be solved… but such was not the case.

[thijs-42]

@Ernest_Chiarello we eventually got it working again.

1. Downgrade the client to 2.4.11.
2. Purge all backups in de backup-server

After we did that. The backups started working again (against newest server )

[Ernest_Chiarello]

@thijs-42 you are right, thanks for the suggestion.
UrBackup Server v2.5.27.0 and UrBackup Client Controller v2.4.11.0 is the winning couple!
Disadvantage of this solution: updates from all other clients are blocked, but we can’t do better until the problem is solved.

[rmm]

Meanwhile I’m on Server 2.5.30. The problem still persists.
I looked into it again, as it seems the problem is the path.
Error saving metadata. Could not open output file at "/var/lib/urbackup/psqlserver/230419-1416/.hashes/urbackup_backup_scripts/postgresbase/.dir_metadata

I’ve taken a look at that directory. “.hashes/urbackup_backup_scripts/postgresbase” is a file. Therefore it can not contain the file .dir_metadata. But in “.hashes/urbackup_backup_scripts” there exists a file .dir_metadata. Probably it wan’t to reference that one?

Anyway, would be cool if there would be a bug tracker somewhere, to follow bugs, and get notified when they are fixed…

[rmm]

just run into the same problem again.
Just out of curriosity.
Did someone find another solution than Client 2.4.11 ?

[Ernest_Chiarello]

January 2024. The urbackup 2.4.11 client is still the only client version for backing up a database with wal_archive. I’m now using version 2.5.32 of the urbackup server. The recent version of the client, 2.5.25, does not work.

thank you in advance to the developers for dealing with this problem.

Ernest.