failed to reexec: Permission denied

[distributed-clouds]

Issue Description

Hello everyone I have built the podman from source on my ubuntu 24.04 on ec2 instance, everything looks good I can pull the images I can run the container, I can build the images. but each time when i reboot the instance or want to run the podman for the first time I get this error
failed to reexec: Permission denied
but when I repeat the command again it works:

podman version

Client:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.22.6
Git Commit:   874bf2c301ecf0ba645f1bb45f81966cc755b7da
Built:        Thu Nov 21 16:21:39 2024
OS/Arch:      linux/amd64

any idea?
thanks for your help

Steps to reproduce the issue

Steps to reproduce the issue

1. run any podman commands for the first time

Describe the results you received

failed to reexec: Permission denied

Describe the results you expected

return normal output

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.1.12, commit: 3d774d57d0ba1857573df331617520688f543444'
  cpuUtilization:
    idlePercent: 83.97
    systemPercent: 6.89
    userPercent: 9.14
  cpus: 2
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: file
  freeLocks: 2048
  hostname: ip-100-68-9-90
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.0-1019-aws
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3503554560
  memTotal: 4023279616
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: Unknown
    path: /usr/local/libexec/podman/netavark
    version: netavark 1.14.0-dev
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.2.0+dev
      commit: v1.2.0-69-gb7da1673
      spec: 1.2.0
      go: go1.22.4
  os: linux
  pasta:
    executable: /usr/local/bin/pasta
    package: Unknown
    version: |
      pasta 2024_11_21.238c69f
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: false
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 0h 1m 44.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/ubuntu/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ubuntu/.local/share/containers/storage
  graphRootAllocated: 30083776512
  graphRootUsed: 6272868352
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/ubuntu/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.0
  Built: 1732206099
  BuiltTime: Thu Nov 21 16:21:39 2024
  GitCommit: 874bf2c301ecf0ba645f1bb45f81966cc755b7da
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[mheon]

Probably related to the rootless pause process, which we both create and try to join on first run, then only join on subsequent runs. You can do a podman system migrate and if that causes the error again that confirms the cause.

[distributed-clouds]

Hi @mheon Thanks for your answer, even for running this command
podman system migrate I get the same error for the first time, and after the second time and rebooting the system, nothing has changed.
BTW, there is no container running on the system