Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Report: long GTIDs cause xtrabackup backups to fail #16303

Closed
rvrangel opened this issue Jul 1, 2024 · 1 comment · Fixed by #16304
Closed

Bug Report: long GTIDs cause xtrabackup backups to fail #16303

rvrangel opened this issue Jul 1, 2024 · 1 comment · Fixed by #16304

Comments

@rvrangel
Copy link
Contributor

rvrangel commented Jul 1, 2024

Overview of the Issue

We reported an issue to Percona where if GTIDs get too long (beacuse we have replaced server enough times in a shard), we get to a point where Xtrabackup starts to truncate the output of its loglines, which means we end up with an invalid/incorrect GTID. This seems to happen the the size of @@gtid_executed gets close to 8KB. Upstream ticket for more details: https://perconadev.atlassian.net/browse/PXB-3302

Reproduction Steps

  1. have a MySQL server with long enough GTIDs. See the Percona ticket for a reproduction of this scenario.
  2. Try to start a backup using the xtrabackupengine.
  3. Backup will fail with:
E0701 04:32:40.512452 3937211 main.go:96] E0701 11:32:40.511911 backup.go:107] E0701 11:32:40.509443 backup.go:127] backup is not usable, aborting it: [Code: INTERNAL
invalid MySQL 5.6 SID "f7e534c2-0f61-2024-07-01T04"

note the T04 is the beginning of the next line of output from xtrabackup.

Binary Version

tested on v14, but this should happen on any version because it is an issue with Xtrabackup and how Vitess obtains the replication position of its backup

Operating System and Environment details

Ubuntu 22.04

Log Fragments

I0701 04:32:40.499083 2078479 xtrabackupengine.go:334] xtrabackup stderr: 2024-07-01T04:32:40.498984-07:00 0 [Note] [MY-011825] [Xtrabackup] completed OK!
I0701 04:32:40.508453 2078479 xtrabackupengine.go:667] Found position: *REDACTED*,f61b69f5-24ae-11ee-864f-126f112aba75:1,f65e62ed-d3b9-11ed-b553-125b93f33a1b:1-2281198,f7e534c2-0f61-2024-07-01T04:32:39.275652-07:000[Note][MY-011825][Xtrabackup]MySQLslavebinlogposition:masterhost
I0701 04:32:40.509243 2078479 xtrabackupengine.go:124] Closing backup file backup.xbstream.gz
E0701 04:32:40.509403 2078479 backup.go:127] backup is not usable, aborting it: [Code: INTERNAL
invalid MySQL 5.6 SID "f7e534c2-0f61-2024-07-01T04"
@rvrangel rvrangel added Type: Bug Needs Triage This issue needs to be correctly labelled and triaged labels Jul 1, 2024
@mattlord mattlord added Component: Backup and Restore and removed Needs Triage This issue needs to be correctly labelled and triaged labels Jul 2, 2024
@mattlord
Copy link
Contributor

mattlord commented Jul 2, 2024

Thank you, @rvrangel ! ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants