container state improper 409 - via stats endpoint for container in created state

[lukasmrtvy]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

• Its not possible to attach to stats API endpoint via /container/{id}/stats if the container is created via /containers/create endpoint and is in created/initialized state. Same applies for CLI (podman stats {id}).
• Its working for Docker 20.10.13

Steps to reproduce the issue:

1. containerid=$(curl -s --unix-socket /run/podman/podman.sock -X POST -H "Content-Type: application/json" -d @payload.json "http://d/containers/create" | jq -r .Id)

2. curl -s --unix-socket /run/podman/podman.sock -X GET http://d/containers/$containerid/stats

Describe the results you received:
Response from step2 {"cause":"container state improper","message":"container state improper","response":409}

Describe the results you expected:
Empty stats are streamed

...
{"read":"0001-01-01T00:00:00Z","preread":"0001-01-01T00:00:00Z","pids_stats":{},"blkio_stats":{"io_service_bytes_recursive":null,"io_serviced_recursive":null,"io_queue_recursive":null,"io_service_time_recursive":null,"io_wait_time_recursive":null,"io_merged_recursive":null,"io_time_recursive":null,"sectors_recursive":null},"num_procs":0,"storage_stats":{},"cpu_stats":{"cpu_usage":{"total_usage":0,"usage_in_kernelmode":0,"usage_in_usermode":0},"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"precpu_stats":{"cpu_usage":{"total_usage":0,"usage_in_kernelmode":0,"usage_in_usermode":0},"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"memory_stats":{},"name":"/pensive_sanderson","id":"3fb3108ac3ed8e0d41062889e6f442bc79b03b73d78f6a93d0a469324acbdabf"}
{"read":"0001-01-01T00:00:00Z","preread":"0001-01-01T00:00:00Z","pids_stats":{},"blkio_stats":{"io_service_bytes_recursive":null,"io_serviced_recursive":null,"io_queue_recursive":null,"io_service_time_recursive":null,"io_wait_time_recursive":null,"io_merged_recursive":null,"io_time_recursive":null,"sectors_recursive":null},"num_procs":0,"storage_stats":{},"cpu_stats":{"cpu_usage":{"total_usage":0,"usage_in_kernelmode":0,"usage_in_usermode":0},"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"precpu_stats":{"cpu_usage":{"total_usage":0,"usage_in_kernelmode":0,"usage_in_usermode":0},"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"memory_stats":{},"name":"/pensive_sanderson","id":"3fb3108ac3ed8e0d41062889e6f442bc79b03b73d78f6a93d0a469324acbdabf"}
...

Additional information you deem important (e.g. issue happens only occasionally):

• Podman CLI returns:

podman stats $containerid
Error: entities.ContainerStatsReport.Error: decode non empty interface: can not unmarshal into nil, error found in #9 byte of ...|{"Error":{},"Stats"|..., bigger context ...|{"Error":{},"Stats":null}

Output of podman version:

Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.18beta2

Built:      Thu Mar  3 14:56:09 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

(paste your output here)host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: coreos
    version: "36"
  eventLogger: journald
  hostname: ip-10-2-17-77
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.5-300.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 13253181440
  memTotal: 32935444480
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 96h 0m 0.13s (Approximately 4.00 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 12
    paused: 0
    running: 3
    stopped: 9
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 26
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319369
  BuiltTime: Thu Mar  3 14:56:09 2022
  GitCommit: ""
  GoVersion: go1.18beta2
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.0.2-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

• AWS
• Fedora CoreOS 36.20220505.3.2

[mheon]

Any chance you have easy access to the Docker CLI and see if they return a similar error to our CLI or just stream all-0s stats like the API does?

[lukasmrtvy]

docker stats pensive_sanderson

CONTAINER ID   NAME                CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
3fb3108ac3ed   pensive_sanderson   0.00%     0B / 0B             0.00%     0B / 0B   0B / 0B     0

[mheon]

@jwhonce If you have time, PTAL; otherwise, could be a good candidate for the interns.

[jakecorrenti]

I can take this one

[rhatdan]

You got it

[jakecorrenti]

Do we want the 0's just like Docker? I have this at the moment after a minor change (before testing)

[rhatdan]

I would guess users would expect this to match Docker as much as possible.