MySQL database not starting

[muhammadhur2]

For some reason I am unable to start MySQL database in Percona Everest. I am however able to start other databases i.e. MongoDB and PostgreSQL.

I have tried with 1 node and 3 nodes and neither works.

These are all the pods in the namespace.

I have added all the logs file in this repo:
Log Files

[oksana-grishchenko]

Hi @muhammadhur2! Thank you for reporting an issue.

To understand the problem better and investigate it, it would be great to have also:

• the MySQL cluster configuration:
  kubectl get pxc <cluster_name_from_everest_UI> -n everest -o yaml
• pxc operator logs:
  kubectl logs -l app.kubernetes.io/name=percona-xtradb-cluster-operator -n everest

[muhammadhur2]

Hello @oksana-grishchenko
I have updated the above log files with the files required. I have, however, hidden the ips used in the yaml file.

This is a new cluster, however, as I had deleted the other one. I still have the same issue though.

[hors]

Hi @muhammadhur2, as I can see you have some communication problems between haproxy and pxc pods. It can be problems with password or network problems:

    Warning  Unhealthy  39m (x18 over 52m)    kubelet            Liveness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
  Warning  BackOff    18m (x9 over 20m)     kubelet            Back-off restarting failed container haproxy in pod mysql-lz4-haproxy-0_everest(70ae1edb-37b9-40a7-b7d4-4bcd836951d6)
  Warning  Unhealthy  14m (x71 over 51m)    kubelet            Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:33062' (111)
  Warning  Unhealthy  4m4s (x464 over 53m)  kubelet            Readiness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2

As you can see https://github.com/percona/percona-docker/blob/main/haproxy/dockerdir/usr/local/bin/liveness-check.sh liveness probe gets password from secret (which is mounted) and use it for connections.

It can be some network problems. Also, it will be nice to see log from pxc-monit container (haproxy). One more question: which platform are you using (EKS, GKE)?