Impossible to override container's DNS with network

[Agalin]

Issue Description

Since podman 4.4.0 it's no longer possible to override DNS entries in resolv.conf for a container if it's connected to a specific network. It leads to connectivity issues due to google dns entries being there.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman network create test
2. podman run --rm -ti --net test --dns 10.0.2.3 fedora cat /etc/resolv.conf

Describe the results you received

search dns.podman .
nameserver 172.18.0.65
nameserver 8.8.8.8
nameserver 8.8.4.4

Describe the results you expected

nameserver 10.0.2.3

podman info output

[fedora@ssoloch-rootless-podman podman-custom]$ podman info
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.1.5, commit: 4cb1e4d73699ce0cef2c3d89b652b3d15be429b3'
  cpuUtilization:
    idlePercent: 99.12
    systemPercent: 0.21
    userPercent: 0.67
  cpus: 4
  distribution:
    distribution: fedora
    variant: cloud
    version: "37"
  eventLogger: journald
  hostname: ssoloch-rootless-podman.novalocal
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.10-200.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 6377996288
  memTotal: 8329482240
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8328835072
  swapTotal: 8328835072
  uptime: 4h 4m 48.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: registry-mirror.apps.k8s.ams.osa
      PullFromMirror: ""
    - Insecure: false
      Location: registry-mirror.apps.k8s.am4.osa
      PullFromMirror: ""
    - Insecure: false
      Location: registry-mirror.apps.k8s.lati.osa
      PullFromMirror: ""
    Prefix: docker.io
    PullFromMirror: ""
  dockerhub.mirror.registry.service.osa:
    Blocked: false
    Insecure: false
    Location: dockerhub.mirror.registry.service.osa
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: registry-mirror.apps.k8s.ams.osa
      PullFromMirror: ""
    - Insecure: false
      Location: registry-mirror.apps.k8s.am4.osa
      PullFromMirror: ""
    - Insecure: false
      Location: registry-mirror.apps.k8s.lati.osa
      PullFromMirror: ""
    - Insecure: false
      Location: docker.io
      PullFromMirror: ""
    Prefix: dockerhub.mirror.registry.service.osa
    PullFromMirror: ""
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/fedora/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/fedora/.local/share/containers/storage
  graphRootAllocated: 52527345664
  graphRootUsed: 5200162816
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/fedora/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1675940333
  BuiltTime: Thu Feb  9 10:58:53 2023
  GitCommit: ""
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Fedora 37 with systemd-resolved enabled.

Additional information

When DNS is configured to the same value in /etc/containers.conf then for 4.3.1 reproduction steps end with:

nameserver 10.0.2.3
nameserver 10.0.2.3

For 4.4.1 it doesn't change anything.

Network created with --dns 10.0.2.3 doesn't change anything, resolv.conf is the same.

Podman is the only package I've found affecting results, downgrading netavark or aardvark doesn't change anything.

[Luap99]

podman 4.4 changed how --dns works on network with dns anabled, it will instead set the dns servers correctly in aardvark-dns which will allow inter container name resolution to work as well as using your custom dns server.

see #16297
I am not sure why it adds 8.8.8.8 in this case but aardvark-dns should use the correct upstream resolver.

[Agalin]

Yeah the aardvark resolver (172.18.0.65 in the example) works correctly. But due to there being Google dns entries resolving generally doesn't work (in 66% of cases...). In the past Google DNS entries were added when resolv.conf was pointing to a localhost resolver (e.g. systemd-resolved like in this case) and --dns was a way to override it. Now I don't see a way to convince podman to stop adding them.

[nivekuil]

Also hit by this breaking change, pods no longer hit local consul for DNS lookups so their nodes are effectively dead. Is the fix just to recreate the network with --disable-dns or is there a less destructive way?

[Luap99]

If you do not use internal dns names then yes --disable-dns is recommended.