Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replication problems when running LHM #138

Open
andrewjhumphrey opened this issue May 3, 2016 · 0 comments
Open

Replication problems when running LHM #138

andrewjhumphrey opened this issue May 3, 2016 · 0 comments

Comments

@andrewjhumphrey
Copy link

Our current setup is
master db server (primarydb) -> RDS instance using RDS external replication (masterreplica) -> RDS read replicas * 2 (us-west-1 and us-west-2)

When we run an LHM on a reasonably busy table on primarydb the read replicas us-west-1 and us-west-2 report errors like this one:

Read Replica Replication Error - SQLError: 1032, reason: Could not execute Update_rows event on table database.table_name; Can't find record in 'table_name', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin-changelog.138461, end_log_pos 1728883

The replication between primarydb and masterreplica continues to work just fine, it's only the read replicas that crack the poops.

All instances use binlog_format = ROW
All instances are configured with log_bin_trust_function_creators = 1
All instances are running 5.6.22 or higher.

Anyone else got this sort of configuration working?

Things we believe but don't have specific evidence to prove (or disprove)

  1. The errors are less likely to occur if no updates occur to the table being LHM'ed (ie the trigger doesn't do any work). LHMs do not always cause replication issues, but it does appear to only not happen on fairly static tables.
  2. Running the LHM on the masterreplica directly does not break the Read replicas, BUT we have not run it on a RDS replica that has updates occuring to it. (ie. it hasn't broken when we've experimented but see above).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant