Replicas are never catch up after pod is recreated

[ynnt]

Hello!
I have an issue with mysql cluster not catching up after failover happens.

Cluster spec:

  spec:
    image: percona:5.7.29-centos
    podSpec:
      metricsExporterResources: {}
      mysqlOperatorSidecarResources: {}
      resources: {}
    replicas: 2
    secretName: my-cluster-secret
    volumeSpec:
      persistentVolumeClaim:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

After I delete mysql-0 pod, mysql-1 starts complaining it can't sync with it:

//my-cluster-mysql-0.mysql.default:3306' - retry-time: 1  retries: 162, Error_code: 2005
2020-08-13T16:37:48.525821Z 939 [ERROR] Slave I/O for channel '': error connecting to master 'sys_replication@//my-cluster-mysql-0.mysql.default:3306' - retry-time: 1  retries: 163, Error_code: 2005
2020-08-13T16:37:49.526148Z 939 [ERROR] Slave I/O for channel '': error connecting to master 'sys_replication@//my-cluster-mysql-0.mysql.default:3306' - retry-time: 1  retries: 164, Error_code: 2005

Even when mysql-0 is recreated and successfully starts replication connection to mysql-1.

I wonder if using a pod name like my-cluster-mysql-0.mysql.default is an issue? Its IP address changes after pod is recreated.
Is it possible to experiment using services per each mysql pod for replication connection?

[AMecea]

Hi @ynnt, When mysql-0 is stopped the mysql-1 is promoted as master and is detached from the old master.

See that the host (sys_replication@//my-cluster-mysql-0.mysql.default:3306) contains a //, in the name, that means that is detached. This is how Orchestrator operates.

My question is does mysql-0 become a replica to mysql-1? If yes then you can ignore all of those errors. Please connect to orchestrator and see there if everything is fine: https://www.presslabs.com/docs/mysql-operator/orchestrator/

[ynnt]

Hey!
It can't become a replica, because it tries to connect using old mysql-1 IP address. Orchestrator reports the replication problem.

show slave status; shows this:
Slave_IO_State: Connecting to master

[ynnt]

Also, are there any ways to start mysql with hostnetwork?

[munjalpatel]

@ynnt @AMecea I am also running into the same issue. @ynnt did you ever managed to resolve this?