Problem with xtrabackup

1

Hi,
We are trying out xrtrabackup for our MySQL servers but are getting the following errors in the log:

2017-12-06 23:56:13: Waiting for file hashing and copying threads…
2017-12-06 23:56:14: ERROR: HT: Error getting free space for path “/backup/urBackup/DB20a/171206-2356/urbackup_backup_scripts/mariadbxtrabackup/brp_366/postalmap.ibd”
2017-12-06 23:56:14: Saving file metadata…
2017-12-06 23:56:15: ERROR: Error writing file metadata to file “/backup/urBackup/DB20a/171206-2356/.hashes/urbackup_backup_scripts/mariadbxtrabackup/brp_365/mailrecipient.ibd”
2017-12-06 23:56:15: ERROR: Error writing file metadata -1
2017-12-06 23:56:15: ERROR: Error saving metadata. Could not open output file at “/backup/urBackup/DB20a/171206-2356/.hashes/urbackup_backup_scripts/mariadbxtrabackup/brp_365/mailrecipient.ibd”
2017-12-06 23:56:15: ERROR: Wrong number of embedded meta-data files. Expected 2319 but got 35
2017-12-06 23:56:15: WARNING: Error removing temporary directory “/backup/urBackup/DB20a/171206-2356/.b68xO+K9SCOF35cLk4Bf9Q”. Directory not empty (code: 39)
2017-12-06 23:56:15: Writing new file list…
2017-12-06 23:56:15: ERROR: Fatal error during backup. Backup not completed
2017-12-06 23:56:15: Transferred 130.014 MB - Average speed: 112.506 MBit/s
2017-12-06 23:56:24: ERROR: FATAL: Backup failed because of disk problems (see previous messages)

The test /usr/local/share/urbackup/scripts/mariadbxtrabackup > /dev/null runs fine but we can’t find any information regarding the errors in the log. Does anyone have any idea?

Regards
Jens

2

Is this with the 2.2.x beta?

A list of files in the tar file would perhaps be helpful.

/usr/local/share/urbackup/scripts/mariadbxtrabackup > tmp.tar
tar tvf tmp.tar
3

Server is 2.1.19 and client 2.1.16 both Ubuntu 16.04. tmp.tar contains the following folders and files:

116_umbrella
brp_365
brp_365_data
brp_366
brp_366_data
brp_367
brp_367_data
mysql
performance_schema
sys
backup-my.cnf
ib_buffer_pool
ibdata1
xtrabackup_binlog_info
xtrabackup_checkpoints
xtrabackup_info
xtrabackup_logfile

/Jens

4

The backup worked fine with just one small database, then we imported 6 more, also small but contains a lot more tables.

We then got an error of to many open files during backup and had to set up open-files-limit and after that we got the error above. Could this be related?

5

A work-around should be running mysql with innodb_file_per_table=0 in my.cnf. That should also reduce the number of open files. See https://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html .

6

Also, I am wondering why e.g. the file brp_366/postalmap.ibd is missing from the tar file list…

7

Sorry, attached are the full structure tmp.tar and all files seems to be there. The open-files error is solved as it by standard was 1024 so that should not be an issue anymore.

"ERROR: HT: Error getting free space for path “/backup/urBackup/DB20a/171206-2356/urbackup_backup_scripts/mariadbxtrabackup/brp_366/postalmap.ibd”
postalmap.ibd is the “largest” table right now (just 20000 rows but all the others have below 1000) it that is any hint

“ERROR: Wrong number of embedded meta-data files. Expected 2319 but got 35”
Is this urbackups internal files or is it an error from xtrabackup

tmp.txt (47.4 KB)

8

No success reproducing this problem yet. Which xtrabackup version are you using?

9

Version 2.4.8

10

Could reproduce with that version. Could you try if it is fixed with that client version: https://www.urbackup.org/downloads/Client/2.1.17/UrBackup%20Client%20Linux%202.1.17.sh ?

11

Yes! It’s working with the 2.1.17 client:

12/11/17 19:31 INFO Transferred 130.014 MB - Average speed: 100.92 MBit/s
12/11/17 19:31 DEBUG Script does not exist urbackup/post_incr_filebackup
12/11/17 19:31 INFO Time taken for backing up client DB20a: 30s
12/11/17 19:31 INFO Backup succeeded

Thanks for quick and great work!

12

Hi
We are having some trouble with xtrabackup again. /usr/local/share/urbackup/scripts/mariadbxtrabackup > /dev/null manually is running fine without errors.

But the log from urbackup is filled with errors, log is attached

urbackup-DB20a.zip (29.3 KB)

We got this problem after upgrading urBackup server to 2.2.11 on ubuntu 16.04. Client is 2.2.5, Mysql server is 5.7 and xtrabackup is upgraded to 2.4.11.

Does anyone have any idea what could be wrong?

/Jens

13

Sorry you are having issues. To solve the problem the client (debug) log would be useful.
If possible, could you post it or send it?

This post describes how to change the client to debug logging, where it is stored and where to send it to if posting is not possible: Having problems with UrBackup? Please read before posting

Thanks!

14

Thanks, attached are the debug log. Pls let me know if there are anything else I could do to help

urbackupclient.zip (42.1 KB)

/Jens

15

Thanks. Could you redo that with this one: https://ssl.webpack.de/beta.urbackup.org/tmp/UrBackupUpdateLinux(10).sh I added some debug messages…

16

Here is the debuglog with the 2.2.6 version
urbackupclient.zip (46.1 KB)

17

Thanks! It may work now with https://ssl.webpack.de/beta.urbackup.org/tmp/UrBackupUpdateLinux(11).sh

18

Sorry, but I still get errors after upgrading the client:
urbackupclient.zip (241.5 KB)

19

Ok. How about this one? https://ssl.webpack.de/beta.urbackup.org/tmp/UrBackupUpdateLinux(12).sh

20

Hi,
Had another go at xtrabackup but still problems. The backup job runs for a while then the client crashes with the following in the kernel log: PipeFile: stdou[12957]: segfault at 7f14e805d000 ip 0000557b7662b1c5 sp 00007f14d3ffeac8 error 6 in urbackupclientbackend[557b7646d000+2f4000]

Attached are the debug log from the client and the serverlog.

Server is Ubuntu 18.04 with UrBackup Server 2.3.8
Client is Ubuntu 16.04 with UrBackup Client 2.3.4
MySQL 5.7.23 and xtrabackup 2.4.15-1

Regards
Jens

urbackup.log (5.7 KB) urbackupclient.log (261.0 KB)