Error with symlinked /etc/resolv.conf

[ivansmm]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When container is run on debian unstable (bookworm) on which /etc/resolv.conf is a symlink to /etc/resolvconf/run/resolv.conf, container startup fails.

Steps to reproduce the issue:

1. podman network create tstnet

2. podman pod create --name tst --hostname tst --network tstnet --publish 1521:1521

3. podman run --pod tst --name dummy alpine

Describe the results you received:
When container is run, the following error appears:

   ERRO[0000] error loading cached network config: network "tstnet" not found in CNI cache
   WARN[0000] falling back to loading from existing plugins on disk
   ERRO[0000] error starting some container dependencies
   ERRO[0000] "error configuring network namespace for container
69ea2ee04b143c908ae5b7bcc4b3404fc7c7d3467fc8e11caf50a36eda76ec8a: error adding pod tst_tst to CNI network \"tstnet\": dnsname
error: dnsmasq failed with \"\\ndnsmasq: directory /etc/resolv.conf for resolv-file is missing, cannot poll\\n\": exit status 5"
Error: error starting some containers: internal libpod error

Describe the results you expected:
Container should start up without errors

Additional information you deem important (e.g. issue happens only occasionally):
If I replace an /etc/resolv.conf symlink by regular file, no error occurs.

Output of podman version:

podman version 3.4.2

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 8
  distribution:
    distribution: debian
    version: unknown
  eventLogger: journald
  hostname: island
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65535
    uidmap:
    - container_id: 0
      host_id: 1007
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65535
  kernel: 5.15.0-2-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 4215324672
  memTotal: 16659128320
  ociRuntime:
    name: runc
    package: 'runc: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2+ds1
      commit: 1.0.2+ds1-2
      spec: 1.0.2-dev
      go: go1.16.9
      libseccomp: 2.5.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1007/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 7528673280
  swapTotal: 7998533632
  uptime: 44h 28m 9.34s (Approximately 1.83 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.logit-ag.de
  - docker.io
store:
  configFile: /home/ivans/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ivans/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 22
  runRoot: /run/user/1007/containers
  volumePath: /home/ivans/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 03:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

ii  podman          3.4.2+ds1-1      amd64        engine to run OCI-based containers in Pods

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[Luap99]

Can you provide the output of podman --log-level debug unshare --rootless-cni ls -l /etc/resolv.conf

[ivansmm]

Here it is:

island:container [master]> podman --log-level debug unshare --rootless-cni ls -l /etc/resolv.conf
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called unshare.PersistentPreRunE(podman --log-level debug unshare --rootless-cni ls -l /etc/resolv.conf) 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/home/ivans/.config/containers/containers.conf" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/ivans/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/ivans/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1007/containers     
DEBU[0000] Using static dir /home/ivans/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp      
DEBU[0000] Using volume path /home/ivans/.local/share/containers/storage/volumes 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/runc"            
INFO[0000] Found CNI network aarnet (type=bridge) at /home/ivans/.config/cni/net.d/aarnet.conflist 
INFO[0000] Found CNI network cni-podman6 (type=bridge) at /home/ivans/.config/cni/net.d/cni-podman6.conflist 
INFO[0000] Found CNI network demnet (type=bridge) at /home/ivans/.config/cni/net.d/demnet.conflist 
INFO[0000] Found CNI network dhlnet (type=bridge) at /home/ivans/.config/cni/net.d/dhlnet.conflist 
INFO[0000] Found CNI network lennox-net (type=bridge) at /home/ivans/.config/cni/net.d/lennox-net.conflist 
INFO[0000] Found CNI network mscnet (type=bridge) at /home/ivans/.config/cni/net.d/mscnet.conflist 
INFO[0000] Found CNI network tstnet (type=bridge) at /home/ivans/.config/cni/net.d/tstnet.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] The path of /etc/resolv.conf in the mount ns is "/etc/resolvconf/run/resolv.conf" 
lrwxrwxrwx 1 nobody nogroup 31 ноя 29 10:39 /etc/resolv.conf -> /etc/resolvconf/run/resolv.conf
DEBU[0000] Cleaning up rootless cni namespace           
DEBU[0000] Called unshare.PersistentPostRunE(podman --log-level debug unshare --rootless-cni ls -l /etc/resolv.conf)

[Luap99]

Ok that looks good, can you show podman unshare --rootless-cni mount and podman unshare --rootless-cni cat /etc/resolv.conf
Is /etc/resolvconf/run/resolv.conf or maybe the directory above another symlink?

[ivansmm]

island:container [master]> podman unshare --rootless-cni mount
/dev/mapper/sys-root on / type ext4 (rw,relatime,errors=remount-ro)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=8112084k,nr_inodes=2028021,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1626868k,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1007 type tmpfs (rw,nosuid,nodev,relatime,size=1626868k,nr_inodes=406717,mode=700,uid=1007,gid=100,inode64)
portal on /run/user/1007/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1007,group_id=100)
tmpfs on /run/user/1007/netns type tmpfs (rw,nosuid,nodev,relatime,size=1626868k,nr_inodes=406717,mode=700,uid=1007,gid=100,inode64)
nsfs on /run/user/1007/netns/cni-da6bcfb7-af15-9257-8501-e25f1d65434b type nsfs (rw)
nsfs on /run/user/1007/netns/rootless-cni-ns type nsfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13789)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/mapper/sys-home on /home type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/)
/dev/mapper/sys-containers on /home/ivans/.local/share/containers type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/)
/dev/mapper/sys-containers on /home/ivans/.local/share/containers/storage/overlay type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/)
overlay on /home/ivans/.local/share/containers/storage/overlay/6269ff100849f527b0084cf00017cb406be5875cd7b748a12cbad4dca7b0142f/merged type overlay (rw,relatime,lowerdir=/home/ivans/.local/share/containers/storage/overlay/l/G5ZLHC2WOKTP2DYZQF43XXYTLL,upperdir=/home/ivans/.local/share/containers/storage/overlay/6269ff100849f527b0084cf00017cb406be5875cd7b748a12cbad4dca7b0142f/diff,workdir=/home/ivans/.local/share/containers/storage/overlay/6269ff100849f527b0084cf00017cb406be5875cd7b748a12cbad4dca7b0142f/work,userxattr)
shm on /home/ivans/.local/share/containers/storage/overlay-containers/b774ab475de0c33112cf58de3406aba9b1f8577d245ff32077b3809895861c1c/userdata/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=64000k,uid=1007,gid=100,inode64)
/dev/mapper/sys-kvm on /kvm type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/)
tmpfs on /run/user/1007/rootless-cni/run/user/1007/netns type tmpfs (rw,nosuid,nodev,relatime,size=1626868k,nr_inodes=406717,mode=700,uid=1007,gid=100,inode64)
nsfs on /run/user/1007/rootless-cni/run/user/1007/netns/cni-da6bcfb7-af15-9257-8501-e25f1d65434b type nsfs (rw)
nsfs on /run/user/1007/rootless-cni/run/user/1007/netns/rootless-cni-ns type nsfs (rw)
tmpfs on /run/user/1007/rootless-cni/run/systemd type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1626868k,mode=755,inode64)
tmpfs on /run/resolvconf/resolv.conf type tmpfs (rw,nosuid,nodev,relatime,size=1626868k,nr_inodes=406717,mode=700,uid=1007,gid=100,inode64)
tmpfs on /var/lib/cni type tmpfs (rw,nosuid,nodev,relatime,size=1626868k,nr_inodes=406717,mode=700,uid=1007,gid=100,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,size=1626868k,nr_inodes=406717,mode=700,uid=1007,gid=100,inode64)
tmpfs on /run/user/1007/netns type tmpfs (rw,nosuid,nodev,relatime,size=1626868k,nr_inodes=406717,mode=700,uid=1007,gid=100,inode64)
nsfs on /run/user/1007/netns/cni-da6bcfb7-af15-9257-8501-e25f1d65434b type nsfs (rw)
nsfs on /run/user/1007/netns/rootless-cni-ns type nsfs (rw)
tmpfs on /run/systemd type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1626868k,mode=755,inode64)

island:container [master]> podman unshare --rootless-cni cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory
island:container [master]> ls -l !$
ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 31 Nov 29 10:39 /etc/resolv.conf -> /etc/resolvconf/run/resolv.conf
island:container [master]> ls -l /etc/resolvconf/run/resolv.conf
-rw-r--r-- 1 root root 385 Dec  1 12:23 /etc/resolvconf/run/resolv.conf
island:container [master]>

/etc/resolvconf/run/resolv.conf is a regular file. Actually I was copying it to /etc/resolv.conf with cp when testing.

[Luap99]

But maybe /etc/resolvconf/run or /etc/resolvconf is a symlink?

[ivansmm]

Yes, /etc/resolvconf/run is a symlink to /run/resolvconf

[Luap99]

If you symlink /etc/resolv.conf directly to /run/resolvconf/resolv.conf it should also work.

I see the problem, I will try to fix it later this week.

[ivansmm]

Thanks! With direct symlink you suggested it works.