[rootless] slirp4netns w/ --disable-host-loopback: host initiated udp communication container <-> host impossible

[BinaryKhaos]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running a UDP-based daemon in a rootless container, makes the daemon essentially useless to the host since communication is one-way (host -> container) because no UDP packets from the container will be delivered to the host due to "--disable-host-loopback" being set for slirp4netns. There is no connection tracking or any kind of differentiation happening within libslirp, it will simply not translate the container address to a proper host address (socket.c: sotranslate_out) if said option is being set.

Steps to reproduce the issue:

1. run a rootless container with dnsmasq (mapping ports appropriately)

2. send a dns request to the container from the host

Describe the results you received:

The request is received and processed on the container but the reply never reaches the host.

Describe the results you expected:

The reply should be received on the host.

Additional information you deem important (e.g. issue happens only occasionally):

An even easier way to reproduce the problem is by simply running netcat on both the container and host with the container listening on e.g. port 53 (udp). Using netcat to communicate with the container from the host, all msgs from the host will be received on the container but not vice-versa.

Output of podman version:

Version:            1.6.3
RemoteAPI Version:  1
Go Version:         go1.13.4
Built:              Thu Nov 28 09:46:26 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.4
  podman version: 1.6.3
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v2
  Conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.2, commit: 65fe0226d85b69fc9e527e376795c9791199153d'
  Distribution:
    distribution: gentoo
    version: unknown
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1065536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1065536
      size: 65536
  MemFree: 27335561216
  MemTotal: 33679798272
  OCIRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.10.6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: TARDIS
  kernel: 5.3.12-191121-r1
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: Unknown
    Version: |-
      slirp4netns version 0.4.1
      commit: 4d38845e2e311b684fc8d1c775c725bfcd5ddc27
  uptime: 1h 13m 15.56s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  ConfigFile: /home/matthew/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.8.0
        fuse-overlayfs: version 0.7
        FUSE library version 3.8.0
        using FUSE kernel interface version 7.31
  GraphRoot: /home/matthew/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/1000
  VolumePath: /home/matthew/.local/share/containers/storage/volumes

[AkihiroSuda]

I think Podman should ditch slirp4netns port forwarder and use RootlessKit port forwarder as in Rootless Docker.
Connections from RootlessKit port forwarder are treated as 127.0.0.1 in the container, and yet
very fast.

The RootlessKit port forwarder can be imported as a Go package: https://github.com/rootless-containers/rootlesskit/blob/master/pkg/port/builtin/builtin.go
Example: https://github.com/rootless-containers/rootlesskit/blob/master/pkg/port/builtin/builtin_test.go

[AkihiroSuda]

I'll try to open PR.

[BinaryKhaos]

Thanks for your great effort and taking the time!

I tried your patch locally, and unfortunately for me, it does not change anything. The behavior is still the same -- except that slirp4netns is no longer used. But udp communication works only from host to container. conntrack also clearly shows an entry but never receives a reply. Is there anything I need to configure or change for this work?

[AkihiroSuda]

Thanks for testing, seems we need to specify local addr explicitly in this dialer?

https://github.com/rootless-containers/rootlesskit/blob/8cf0679be24c640267784f500c65ace2b44b0412/pkg/port/builtin/builtin.go#L462

[AkihiroSuda]

hmm, local addr is already set to 127.0.0.1 as expected, but UDP reply seems not forwarded

[smekkley]

I'm wondering. Would that be possible maybe in the future to support packet filtering per container with rootlesskit for outgoing traffic? For example it would be useful if one container can access internet, while other containers can't on the same host. With rootful container is just attaching device and forward iptables, i wonder if it's even worth considering supporting a such feature for rootkit.

[AkihiroSuda]

You can already run iptables in rootlesskit (Not rootkit! 😄 )

[baude]

you can also limit container network access in a variety of other methods including setting up cni networks that do not forward past the host.

[smekkley]

@AkihiroSuda
I'm curious how it works including how it keep persistence. I guess it's not technically iptables (filtering doesn't happen in kernel space), or may be it requires some new kernel feature?

[smekkley]

@baude
Thanks a lot! I didn't know I can use cni with rootless podman. For now, I just did removed default gateway and added only necessary destinations.

[mheon]

I don't believe you can. We explicitly do not initialize CNI as rootless because it requires too many permissions to use.

[smekkley]

Indeed. It failed. I believe the only way to filter in the userspace is by implementing some sort of tcp/udp filter.