Podman fails to recognize HEALTHCHECK in certain BuildKit images

[0xF4CED]

Issue Description

When pulling (some) images that were built with BuiltKit, podman fails to parse/store the declared healtcheck configuration of the dockerimage correctly. Tested with docker.io/adguard/adguardhome@sha256:fc01fa555e6b324378176dff676c7791b2bb6ebfcf63bd2599db568287de7230
This appears to be (almost) identical to the previously resolved issue #12226, possibly indicating a regression or incomplete fix.

Steps to reproduce the issue

podman pull docker.io/adguard/adguardhome@sha256:fc01fa555e6b324378176dff676c7791b2bb6ebfcf63bd2599db568287de7230
podman run --privileged --rm --name dinp --replace -d docker.io/docker
podman exec dinp docker pull docker.io/adguard/adguardhome@sha256:fc01fa555e6b324378176dff676c7791b2bb6ebfcf63bd2599db568287de7230
diff --color <(podman image inspect adguard/adguardhome | grep -i -A3 healthcheck) <(podman exec dinp docker inspect adguard/adguardhome | grep -A8 Healthcheck)

Describe the results you received

<                     "created_by": "HEALTHCHECK &{[\"CMD\" \"/opt/adguardhome/scripts/healthcheck.sh\"] \"30s\" \"10s\" \"0s\" '\\x03'}",
<                     "comment": "buildkit.dockerfile.v0",
<                     "empty_layer": true
<                },
---
>             "Healthcheck": {
>                 "Test": [
>                     "CMD",
>                     "/opt/adguardhome/scripts/healthcheck.sh"
>                 ],
>                 "Interval": 30000000000,
>                 "Timeout": 10000000000,
>                 "Retries": 3
>             },

Describe the results you expected

Podman should properly parse and store the healthcheck that is specified in the dockerimage.

podman info output

Details

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 81.04
    systemPercent: 6.34
    userPercent: 12.63
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: silverblue
    version: "38"
  eventLogger: journald
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.3.7-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1171042304
  memTotal: 8250187776
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8249143296
  swapTotal: 8249143296
  uptime: 0h 8m 56.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/podmanuser/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/podmanuser/.local/share/containers/storage
  graphRootAllocated: 498403901440
  graphRootUsed: 44006096896
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 60
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/podmanuser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685123928
  BuiltTime: Fri May 26 19:58:48 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

When pulling the image with docker, saving the image and loading it into podman, everything works as expected.

podman exec dinp docker save adguard/adguardhome | podman load
podman inspect adguardhome:latest | grep -A8 Healthcheck

"Healthcheck": {
               "Test": [
                    "CMD",
                    "/opt/adguardhome/scripts/healthcheck.sh"
               ],
               "Interval": 30000000000,
               "Timeout": 10000000000,
               "Retries": 3
          }

Image details:

adguardhome Dockerfile , adguardhome buildscript

HEALTHCHECK \
	--interval=30s \
	--timeout=10s \
	--retries=3 \
	CMD [ "/opt/adguardhome/scripts/healthcheck.sh" ]

[rhatdan]

Did you store it in Docker format? I am not sure HEALTHCHECK is stored in OCI Format?

[0xF4CED]

HEALTHCHECK is only a part of the Docker image spec and not OCI. The image in my example is in the Docker format.
#12239 introduced support for health checks from image configurations. However, in certain cases, podman pull fails to parse or store the health check configuration.
I think the problem is easier to understand when following the steps in Steps to reproduce and Additional information, that I provided above.

[flouthoc]

@salevdns As far as I can see #12239 only corrected to get Imageconfig for images when it is in docker format, and I think it works in case of podman exec dinp docker save adguard/adguardhome | podman load cause docker save is saving in doceker format.

Is expectation here is to check if image metadata somehow contains Healthcheck if yes, include it in OCI's image's reserved field Healthcheck and podman must implement it while running the image ?

[0xF4CED]

I am not sure tbh. To me it looks like it should be stored in the docker format when pulling.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc @iamkirkbater Any update on this?

[olifre]

In case it helps, I created a minimal test container in my accidentally reported duplicate issue:

• https://hub.docker.com/r/olifre/podman-healthcheck-test
  ( https://github.com/olifre/podman-healthcheck-test/blob/main/Dockerfile )

I observed the same problem pulling that one, and also the official iobroker container which is built from this Dockerfile. Pulling the same containers with Docker and running them there reveals a working healthcheck.

[FyiurAmron]

FWIW, I started getting this with Kaniko some time ago for no apparent reason. Older versions of Kaniko+podman had healthchecks working normally, and, as expected, Kaniko+docker works without any problems. Forced me to switch back to docker for the time being :/