On heavy host activity mysql sometimes never bootstraps

[gboutry]

Steps to reproduce

Not easily reproducible, but I've seen it occur multiple times

1. Have timeout?

Expected behavior

Mysql bootstraps correctly

Actual behavior

Mysql fails to bootstrap, visible error in logs:

Shell.connect: Access denied for user \'serverconfig\'@\'mysql-0.mysql-endpoints.openstack.svc.cluster.local\'

Versions

Juju CLI: 3.6
mysql.log

Juju agent: 3.6.0

Charm revision: 8.0/stable 180

k8s: v1.31.3 1698

Log output

It fails towards the beginning:

unit-mysql-0: 13:58:01 INFO juju.worker.uniter found queued "leader-elected" hook
unit-mysql-0: 13:58:06 WARNING unit.mysql/0.juju-log Failed to check if cluster metadata exists from_instance='mysql-0.mysql-endpoints.openstack.svc.cluster.local'
unit-mysql-0: 13:58:09 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-mysql-0: 13:58:10 INFO unit.mysql/0.juju-log Setting up the logrotate configurations
unit-mysql-0: 14:00:31 ERROR unit.mysql/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 888, in <module>
    main(MySQLOperatorCharm)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 551, in main
    manager.run()
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 530, in run
    self._emit()
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 519, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 701, in _on_mysql_pebble_ready
    self._configure_instance(container)
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 611, in _configure_instance
    container.restart(MYSQLD_SAFE_SERVICE)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 2274, in restart
    self._pebble.restart_services(service_names)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 2201, in restart_services
    return self._services_action('restart', services, timeout, delay)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 2221, in _services_action
    resp = self._request('POST', '/v1/services', body=body)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 1995, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 2034, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/pebble.py", line 373, in http_open
    return self.do_open(
  File "/usr/lib/python3.10/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out

Then never succeeds to login as serverconfig

Additional context

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6145.

This message was autogenerated