Implement HEALTHCHECK instruction #94

iBobik · 2017-01-11T14:41:06Z

There is a new Dockerfile instruction HEALTHCHECK what is useful for detecting if container is still starting (and other should wait) or unhealthy.

It it enought to write something like this?

HEALTHCHECK CMD mysql --user=$MYSQL_USER --password=$MYSQL_PASSWORD -e "show databases;"

The text was updated successfully, but these errors were encountered:

iBobik · 2017-01-11T14:52:05Z

Maybe to make it working also if MYSQL_USER or MYSQL_PASSWORD is not filled → use root. Then if MYSQL_ALLOW_EMPTY_PASSWORD and other related settings.

Any ideas?

tianon · 2017-01-11T18:35:34Z

docker-library/cassandra#76 (comment) has some good points regarding adding default health checks to official images 👍

(https://github.com/docker-library/healthcheck also has copies of several images with reasonably-generic HEALTHCHECK instructions added)

yosifkit · 2017-07-20T23:52:25Z

Current stance is still the same as linked above in cassandra.

I do not feel that generalized healthchecks on the official images are really that useful.

users will have their own idea of what is "healthy"

it does not actually test that the service is listening to connections outside of localhost (see https://github.com/docker-library/healthcheck for some examples that do more than what's proposed here, including attempting to check whether the service is listening remotely)

some of the Official Images even purposely start in a localhost only mode for database initialization and then kill and start the main service with full network availability

after upgrading their images, current users will have extra unexpected load on their systems for healthchecks they don't necessarily need/want and may be unaware of

Closing old issue.

SalathielGenese · 2018-10-08T21:42:18Z

This' how is made it

mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]

ebuildy · 2019-05-22T12:23:23Z

Pay attention, if you use the /docker-entrypoint-initdb.d folder to run some SQL scripts at runtime, it may be useful to test if data is ready instead of just the server.

katsar0v · 2020-06-18T16:22:17Z

Pay attention, if you use the /docker-entrypoint-initdb.d folder to run some SQL scripts at runtime, it may be useful to test if data is ready instead of just the server.

That's an issue. Can we reopen this issue? If you have entrypoint sql or sh, the healthcheck from @SalathielGenese does not work

grooverdan · 2021-05-12T14:36:17Z

ideas welcome - https://jira.mariadb.org/browse/MDEV-25434

Ajedi32 · 2022-03-16T18:30:31Z

FYI, it looks like there's now a healthcheck script built into the image, it's just not called in the Dockerfile for some reason. You can enable it like so:

services:
  mariadb:
     environment:
       MARIADB_MYSQL_LOCALHOST_USER: true
     healthcheck:
       test: ["CMD", "/usr/local/bin/healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized"]
  other_service:
    depends_on:
      mariadb:
        condition: service_healthy

For more details, and additional tests that can be enabled, see https://github.com/MariaDB/mariadb-docker/blob/master/healthcheck.sh

IMO this (or something similar) should be enabled by default. I'm a firm believer in setting reasonable defaults, and I don't think external services failing to start because docker-compose has no way of checking whether the service is healthy qualifies as "reasonable".

users will have their own idea of what is "healthy"

Users can customize the health check command if they want to. That's not a good reason for the default behavior to be "no health checks at all; just fail silently and don't give the container engine any indication that the service is unhealthy".

it does not actually test that the service is listening to connections outside of localhost

Okay. Again, that's not an argument for not having a basic healthcheck by default. If users want something more comprehensive than what Docker's internal health check mechanism can provide then they can always write it themselves.

some of the Official Images even purposely start in a localhost only mode for database initialization and then kill and start the main service with full network availability

All the more reason for those official images to have a proper health check predefined, so users don't try to write their own health check and discover that behavior the hard way. In images where it's applicable the default health check should detect "localhost only mode" and react appropriately. If programing that is too much effort for some reason, then leave out the health check on the images where that's a problem, and include a note about why. That'd be disappointing, but no worse than the current status quo.

after upgrading their images, current users will have extra unexpected load on their systems for healthchecks they don't necessarily need/want and may be unaware of

It's a healthcheck, not a stress test. A simple healthcheck script running twice a minute should not impose any significant additional load on a system. If for whatever reason the user wants something more complicated that is resource intensive, then let them enable that themselves so they're aware of the trade-off.

Luc45 · 2022-03-30T13:54:01Z

This' how is made it

mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]

This is wrong, please edit your reply to not induce others into error.

If you do this, it will do the ping with root user with an empty password, which can fail (silently) with this message:

root@b8ab90e765c2:/# mysqladmin ping       
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: NO)'
root@b8ab90e765c2:/# echo $?
0

As you can see, the access was denied, but the exit status code was still 0!

You need to specify the root password to use that command. For instance, if your root password is 123:

healthcheck:
    test: ["CMD-SHELL", "mysqladmin ping -P 3306 -p123 | grep 'mysqld is alive' || exit 1"]
    interval: 2s
    retries: 15

Ajedi32 · 2022-03-30T14:03:36Z

@Luc45 Ping just checks the connection, not whether your credentials are correct. It will exit with 1 if the server is not running, which might be good enough depending on your goal. If you need something more complicated than that, see the example in my previous comment.

jerdoe · 2022-05-08T21:23:52Z

FYI, it looks like there's now a healthcheck script built into the image, it's just not called in the Dockerfile for some reason. You can enable it like so:
services:
  mariadb:
     environment:
       MARIADB_MYSQL_LOCALHOST_USER: true
     healthcheck:
       test: ["CMD", "/usr/local/bin/healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized"]
  other_service:
    depends_on:
      mariadb:
        condition: service_healthy
For more details, and additional tests that can be enabled, see https://github.com/MariaDB/mariadb-docker/blob/master/healthcheck.sh

When I use these settings, the healthcheck test result for option --connect is correctly reported but I get a warning in my logs every time the healthcheck.sh script is run :

root@0fe582a68cc7:/# healthcheck.sh --su-mysql --connect
root@0fe582a68cc7:/# echo $?
0

$ sudo docker compose logs -f mariadb
proj_mariadb-1  | 2022-05-08 21:10:02 1453 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:10:32 1461 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:11:03 1463 [Warning] Access denied for user 'mysql'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:11:05 1443 [Warning] Access denied for user 'mysql'@'127.0.0.1' (using password: NO)

Samely, if I enter the following command, the same warning is thrown in my docker compose logs and in my console :

root@0fe582a68cc7:/# gosu mysql mysql -h localhost --protocol tcp -e 'select 1'
ERROR 1045 (28000): Access denied for user 'mysql'@'127.0.0.1' (using password: NO)

but this would not throw the warning:

root@0fe582a68cc7:/# gosu mysql mysql -h localhost -e 'select 1'
+---+
| 1 |
+---+
| 1 |
+---+

Is that something which is expected or is there something wrong with my setup ?

grooverdan · 2022-05-08T22:29:43Z

MARIADB_MYSQL_LOCALHOST_USER creates mysql@localhost but the --connect is using a tcp connection so mysql@127.0.0.1 is the expected user. Tested locally and mysql@::1 was used.

Its possible to hide these with log_warnings=1.

I maybe think I'll just expand the MARIADB_MYSQL_LOCALHOST_USER to create the mysql@127.0.0.1/ mysql@::1 user. I hope it's not too confusing to have MARIADB_MYSQL_LOCALHOST_GRANTS not give the same grants on those users.

Thanks for your comment @jerdoe. Nothing is really wrong with your setup. Creating new issues is ok too.

orefalo · 2022-06-01T15:04:03Z

I too see

│ 2022-06-01 15:02:47 215 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:02:57 216 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:02:57 217 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:07 218 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:07 219 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:17 220 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:17 221 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:27 222 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:27 223 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:37 224 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

on the logs, and ended up on this thread.

grooverdan · 2022-06-01T15:15:37Z

yep, started working on it in #430, I haven't overcome all cases yet, for the moment --log-warnings=1 can hide them.

orefalo · 2022-06-01T15:47:47Z

Yes, saw it - thank you.
I can live with the Warnings, now that I know it's a non issue ;-)

Luc45 · 2022-06-01T16:54:05Z

For whoever it may concern, I ended up doing it like this:

services:
  db:
    healthcheck:
      test: ["CMD-SHELL", "mysql dbname -udbuser -pdbpass -e 'SELECT 1;'  || exit 1"]
      interval: 2s
      retries: 120
  php:
    depends_on:
      db:
        condition: service_healthy

The healthcheck will only pass when a SELECT has successfully ran against the DB, which solved all racing conditions for me of the DB not being ready, even when the ping was successful.

grooverdan · 2022-06-01T22:48:13Z

@Luc45 watch out, the initialization starts a temporary server on initialization and upgrades. Its possible this healthcheck will succeed when its in a getting ready stage. --protocol tcp will save you however there is....

While I haven't implemented user and pass on a command line, there are configuration files that can be passed that can contain:

[mariadb-client]
user=
password=
database=

And then:
test: ["CMD-SHELL", "healthcheck.sh --defaults-file=/etc/mysql/healthcheck.cnf-notdefault" --connect]

monkeyhybrid · 2022-06-02T11:50:59Z

@grooverdan Thanks for that previous comment, suggesting the use of a custom healthcheck.cnf with the relevant auth credentials and using that instead of the mysql@localhost user (which as well as access denied warnings, also failed to ever pass the --innodb_initialized test for me).

I can now use the following healthcheck command and it works perfectly for both tests with no warnings.

healthcheck:
      test: ["CMD", "/usr/local/bin/healthcheck.sh", "--defaults-file=/etc/mysql/healthcheck.cnf-notdefault", "--connect", "--innodb_initialized"]

With this, there is also no need to set the MARIADB_MYSQL_LOCALHOST_USER environment variable.

Thanks!

PrynsTag · 2022-09-14T15:46:54Z

If you have secrets and environment variables, you could also do this.

services:
  db:
    environment:
        FILE__MYSQL_ROOT_PASSWORD: /run/secrets/mysql_root_password
    secrets:
        - mysql_root_password
    healthcheck:
      test: test: ["CMD-SHELL", "mysqladmin ping -P $$(echo $$MARIADB_PORT) -p$$(cat $$FILE__MYSQL_ROOT_PASSWORD) | grep 'mysqld is alive' || exit 1"]
      interval: 2s
      retries: 120
  php:
    depends_on:
      db:
        condition: service_healthy

GagMirz · 2023-02-02T11:11:43Z

If you have secrets and environment variables, you could also do this.

services:
  db:
    environment:
        FILE__MYSQL_ROOT_PASSWORD: /run/secrets/mysql_root_password
    secrets:
        - mysql_root_password
    healthcheck:
      test: test: ["CMD-SHELL", "mysqladmin ping -P $$(echo $$MARIADB_PORT) -p$$(cat $$FILE__MYSQL_ROOT_PASSWORD) | grep 'mysqld is alive' || exit 1"]
      interval: 2s
      retries: 120
  php:
    depends_on:
      db:
        condition: service_healthy

Thanks, this one was the only working option on my side.

grooverdan · 2023-02-02T23:05:19Z

ack. Yes I do need to do something better.

Yszty · 2023-02-27T11:10:24Z

This' how is made it
mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]
This is wrong, please edit your reply to not induce others into error.

If you do this, it will do the ping with root user with an empty password, which can fail (silently) with this message:
root@b8ab90e765c2:/# mysqladmin ping       
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: NO)'
root@b8ab90e765c2:/# echo $?
0
As you can see, the access was denied, but the exit status code was still 0!

You need to specify the root password to use that command. For instance, if your root password is 123:
healthcheck:
    test: ["CMD-SHELL", "mysqladmin ping -P 3306 -p123 | grep 'mysqld is alive' || exit 1"]
    interval: 2s
    retries: 15

MySQL Documentation says about mysqladmin ping

Check whether the server is available. The return status from mysqladmin is 0 if the server is running, 1 if it is not. This is 0 even in case of an error such as Access denied, because this means that the server is running but refused the connection, which is different from the server not running.

gremo · 2023-05-14T18:58:30Z

@grooverdan @monkeyhybrid can you elaborate your answer a bit more? Same problem here, lot of warnings due to the healthcheck script.

Kamikaze01git · 2023-05-15T09:32:46Z

If i use

healthcheck:
      test: ["CMD-SHELL", "mysqladmin ping -P 3306 -p MyRootPW | grep 'mysqld is alive' || exit 1"]
      interval: 2s
      retries: 15

just to ping my db, i got status unhealthy with log
Segmentation fault (core dumped)

any help?

saschabrockel · 2023-06-14T01:22:46Z

Using MariaDB 11.0.2 in Docker I am very curious that nothing mentioned here works. If I try to execute healthcheck.sh --su-mysql --connect --innodb_initialized in the container I get the following error: failed switching to "mysql": operation not permitted

I have set MARIADB_MYSQL_LOCALHOST_USER to true and also tried to set it to 1. Does not change anything. In the container the commands like mysql or mysqladmin are not found and unknown.

And last but not least:
healthcheck.sh --defaults-file=/etc/mysql/healthcheck.cnf-notdefault --connect --innodb_initialized Could not open required defaults file: /etc/mysql/healthcheck.cnf-notdefault
Fatal error in defaults handling. Program aborted
healthcheck innodb_initialized failed

Is something wrong with my permissions?

grooverdan · 2023-06-14T08:49:46Z

Like the SO, I don't know what causes "failed switching to "mysql": operation not permitted". More information on the OS / Docker version might be applicable here.

MARIADB_MYSQL_LOCALHOST_USER doesn't create mysql/mysqladmin links.

"Could not open required defaults file: /etc/mysql/healthcheck.cnf-notdefault" - looks like you haven't passed this file into the container as a volume.

I was updating https://mariadb.com/kb/en/using-healthcheck-sh-script/ today and drafting a blog about the removal of mysql/mysqladmin that was in the release notes.

Please create a new issue for bugs. Keeping track of old threads where the situation has changed significantly is difficult.

Yszty · 2023-06-14T08:58:26Z

Yesterday I rebuilt my containers and health check also stopped work. The problem was that on latest mariadb containers command mysql does not work,(read here #512) after change to mariadb, everything is fine in my case.

Doesnt work:
health-check.sh

#!/bin/sh

mysql --user=root --password=$(cat $MARIADB_ROOT_PASSWORD_FILE) --silent --execute "SELECT 1"
exit $?;

Fix:
health-check.sh

#!/bin/sh

mariadb --user=root --password=$(cat $MARIADB_ROOT_PASSWORD_FILE) --silent --execute "SELECT 1"
exit $?;

docker-compose.yml

  db:
    platform: linux/amd64
    build:
      context: ./mariadb
    ports:
      - '3306:3306'
    restart: always
    healthcheck:
      test: health-check.sh
      timeout: 30s
      retries: 10
      interval: 2s

Dockerfile

FROM mariadb
COPY health-check.sh /bin/health-check.sh
RUN chmod +x /bin/health-check.sh

grooverdan · 2023-06-14T09:12:40Z

see #512 and https://mariadb.org/mariadb-server-docker-official-images-healthcheck-without-mysqladmin/

buggsi · 2024-02-08T04:45:07Z

Set the MYSQL_ROOT_PASSWORD (or MARIADB_ROOT_PASSWORD) variable in a .env file and add this to the mariadb service in docker-compose.yml:

    healthcheck:
      test: mariadb-admin ping -h 127.0.0.1 --password=$MYSQL_ROOT_PASSWORD
      start_period: 20s
      start_interval: 1s
      interval: 30s
      timeout: 5s
      retries: 30

Edamijueda · 2025-01-19T00:15:32Z

I resolve this issue by using a GA(General Availability) version of mariadb. In my case image: mariadb:11.6.2

healthcheck:
      test: [ "CMD", "healthcheck.sh", "--connect", "--innodb_initialized" ]
      start_period: 1m
      start_interval: 10s
      interval: 1m
      timeout: 5s
      retries: 3

HealthCheck is now healthy.

yosifkit closed this as completed Jul 20, 2017

yosifkit mentioned this issue Sep 20, 2017

Add a HEALTHCHECK stanza #128

Closed

tgross35 mentioned this issue Feb 26, 2022

Suggested improvements to healthcheck - user / password from environment variables #416

Closed

grooverdan mentioned this issue May 9, 2022

Excessive error log warnings with healthcheck --connect #430

Closed

larueli mentioned this issue Feb 22, 2023

Chore/docker ungdev/etuutt-api#41

Merged

4 tasks

jbergstroem mentioned this issue Mar 9, 2023

feat: use mariadb client for healthchecks jbergstroem/mariadb-alpine#221

Draft

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement HEALTHCHECK instruction #94

Implement HEALTHCHECK instruction #94

iBobik commented Jan 11, 2017

iBobik commented Jan 11, 2017 •

edited

Loading

tianon commented Jan 11, 2017

yosifkit commented Jul 20, 2017

SalathielGenese commented Oct 8, 2018

ebuildy commented May 22, 2019

katsar0v commented Jun 18, 2020

grooverdan commented May 12, 2021

Ajedi32 commented Mar 16, 2022 •

edited

Loading

Luc45 commented Mar 30, 2022 •

edited

Loading

Ajedi32 commented Mar 30, 2022 •

edited

Loading

jerdoe commented May 8, 2022 •

edited

Loading

grooverdan commented May 8, 2022 •

edited

Loading

orefalo commented Jun 1, 2022

grooverdan commented Jun 1, 2022

orefalo commented Jun 1, 2022

Luc45 commented Jun 1, 2022

grooverdan commented Jun 1, 2022 •

edited

Loading

monkeyhybrid commented Jun 2, 2022

PrynsTag commented Sep 14, 2022 •

edited

Loading

GagMirz commented Feb 2, 2023

grooverdan commented Feb 2, 2023

Yszty commented Feb 27, 2023

gremo commented May 14, 2023

Kamikaze01git commented May 15, 2023

saschabrockel commented Jun 14, 2023

grooverdan commented Jun 14, 2023

Yszty commented Jun 14, 2023 •

edited

Loading

grooverdan commented Jun 14, 2023

buggsi commented Feb 8, 2024 •

edited

Loading

Edamijueda commented Jan 19, 2025

Implement HEALTHCHECK instruction #94

Implement HEALTHCHECK instruction #94

Comments

iBobik commented Jan 11, 2017

iBobik commented Jan 11, 2017 • edited Loading

tianon commented Jan 11, 2017

yosifkit commented Jul 20, 2017

SalathielGenese commented Oct 8, 2018

ebuildy commented May 22, 2019

katsar0v commented Jun 18, 2020

grooverdan commented May 12, 2021

Ajedi32 commented Mar 16, 2022 • edited Loading

Luc45 commented Mar 30, 2022 • edited Loading

Ajedi32 commented Mar 30, 2022 • edited Loading

jerdoe commented May 8, 2022 • edited Loading

grooverdan commented May 8, 2022 • edited Loading

orefalo commented Jun 1, 2022

grooverdan commented Jun 1, 2022

orefalo commented Jun 1, 2022

Luc45 commented Jun 1, 2022

grooverdan commented Jun 1, 2022 • edited Loading

monkeyhybrid commented Jun 2, 2022

PrynsTag commented Sep 14, 2022 • edited Loading

GagMirz commented Feb 2, 2023

grooverdan commented Feb 2, 2023

Yszty commented Feb 27, 2023

gremo commented May 14, 2023

Kamikaze01git commented May 15, 2023

saschabrockel commented Jun 14, 2023

grooverdan commented Jun 14, 2023

Yszty commented Jun 14, 2023 • edited Loading

grooverdan commented Jun 14, 2023

buggsi commented Feb 8, 2024 • edited Loading

Edamijueda commented Jan 19, 2025

iBobik commented Jan 11, 2017 •

edited

Loading

Ajedi32 commented Mar 16, 2022 •

edited

Loading

Luc45 commented Mar 30, 2022 •

edited

Loading

Ajedi32 commented Mar 30, 2022 •

edited

Loading

jerdoe commented May 8, 2022 •

edited

Loading

grooverdan commented May 8, 2022 •

edited

Loading

grooverdan commented Jun 1, 2022 •

edited

Loading

PrynsTag commented Sep 14, 2022 •

edited

Loading

Yszty commented Jun 14, 2023 •

edited

Loading

buggsi commented Feb 8, 2024 •

edited

Loading