podman-compose and rootless container DNS names resolution problem

[duck-rh]

Issue Description

Quack,

I am running a compose configuration which is a templated and slightly fixed version from the docker-mailman project. The three containers start but accessing the database and mailman-core REST API fails from the mailman-web container.

It looks a bit similar to #16939 but I'm not really sure this is the same problem.

Steps to reproduce the issue

Compose file (with secrets templated):

version: '2'

services:
  mailman-core:
    image: ghcr.io/maxking/mailman-core:0.4
    container_name: mailman-core
    hostname: mailman-core
    volumes:
      - /opt/mailman/core:/opt/mailman/:Z
    stop_grace_period: 30s
    depends_on:
      - database
    environment:
      DATABASE_URL: "postgresql://mailman:{{ _mailman_db_pwd }}@database/mailmandb"
      DATABASE_TYPE: postgres
      DATABASE_CLASS: mailman.database.postgresql.PostgreSQLDatabase
      HYPERKITTY_API_KEY: "{{ _hyperkitty_api_key }}"
      MTA: postfix
    ports:
      - "127.0.0.1:8001:8001" # API
      - "127.0.0.1:8024:8024" # LMTP - incoming emails
    networks:
      mailman:

  mailman-web:
    image: ghcr.io/maxking/mailman-web:0.4
    container_name: mailman-web
    hostname: mailman-web
    depends_on:
      - database
    volumes:
      - /opt/mailman/web:/opt/mailman-web-data:Z
    environment:
      DATABASE_URL: "postgresql://mailman:{{ _mailman_db_pwd }}@database/mailmandb"
      DATABASE_TYPE: postgres
      SECRET_KEY: "{{ _django_secret_key }}"
      UWSGI_STATIC_MAP: "/static=/opt/mailman-web-data/static"
      SERVE_FROM_DOMAIN: "{{ webui_vhost }}"
      HYPERKITTY_API_KEY: "{{ _hyperkitty_api_key }}"
    ports:
      - "127.0.0.1:8000:8000" # HTTP
      - "127.0.0.1:8080:8080" # uwsgi
    networks:
      mailman:

  database:
    image: docker.io/library/postgres:11-alpine
    container_name: database
    hostname: database
    volumes:
      - /opt/mailman/database:/var/lib/postgresql/data:Z
    environment:
      POSTGRES_DB: mailmandb
      POSTGRES_USER: mailman
      POSTGRES_PASSWORD: "{{ _mailman_db_pwd }}"
    networks:
      mailman:
    restart: always

networks:
   mailman:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.19.199.0/24

• install podman podman-plugins containernetworking-plugins crun fuse-overlayfs slirp4netns podman-compose
• create a local user
• su/sudo
• podman-compose -f container-compose.yml up

Describe the results you received

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "database" to address: Name does not resolve

Describe the results you expected

DNS resolution with container names works.

podman info output

The machine is running CentOS Stream 9.

podman info:

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.6-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: ae34694dbfd15d1b90e7219e4a964df2f1c6c415'
  cpuUtilization:
    idlePercent: 97.78
    systemPercent: 1.25
    userPercent: 0.98
  cpus: 2
  distribution:
    distribution: '"centos"'
    version: "9"
  eventLogger: file
  hostname: lists.osci.io
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000000
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000000
  kernel: 5.14.0-283.el9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 122851328
  memTotal: 1692098560
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.1-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 6h 29m 4.00s (Approximately 0.25 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /opt/mailman/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /opt/mailman/.local/share/containers/storage
  graphRootAllocated: 21464350720
  graphRootUsed: 1059188736
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /opt/mailman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.0
  Built: 1675439377
  BuiltTime: Fri Feb  3 10:49:37 2023
  GitCommit: ""
  GoVersion: go1.19.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.0

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

_No response_

### Additional information

I went into the mailman-web container to have a look and checked `/etc/resolv.conf`:

search dns.podman osci.io redhat.com
nameserver 10.89.0.1
nameserver 8.43.85.229
nameserver 8.43.85.230
options edns0 rotate

I was surprised to see my host configuration but it seems that's what docker does and that's common. Now in a rootless container DNS is run by the new netavark stack IIUC, and it seems to be running fine:

_mailman 22571 1 0 05:26 ? 00:00:00 /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run

So why would we need to have the host DNS configuration knowing that our DNS servers have no knowledge of these containers? Moreover because we're using the `rotate` option we're sure to hit problems.

I removed this option but that did not solve the problem entirely. Testing manually the script that is supposed to work in this container it still failed most of the time. Only after I removed our two nameservers to only keep the netavark one did it worked each time I tried.

[Luap99]

I think this is the same issue as #17499 which was fixed in #17578

[duck-rh]

@Luap99 indeed that looks like it, thanks.

Not sure who's in charge of the packaging but if one of you is It would be nice to have the fix propagated to Stream 9.

[Luap99]

I am not sure if we plan more 4.4.x releases upstream.

For centos/rhel you should create bugzillas and request backports. I think the RHEL release is already frozen so we need to get extra approval, not sure how it works in centos stream.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[duck-rh]

Thanks @github-actions, that's very helpful.

I'm not sure in which release this fix is included so I've not requested a backport yet. I'll try to check that soon and report here when I am able to test.

[Luap99]

I think this is fixed in 4.4.3 https://github.com/containers/podman/releases/tag/v4.4.3