Rootless, bridge network and reaching a host service from a container

[andreaippo]

Hi,

I am struggling with the following:

I have a compose file with a bunch of services (Kafka, Mongo, etc) and running on a custom bridge network I created manually (let's call it my-bridge).

I then have a docker file that builds and runs a Java webapp (my-service-a) on my-bridge:

[...]
docker run -p 8090:8080 --name=my-service-a --replace --net=my-bridge_default --add-host=host.docker.internal:host-gateway -d org.me/my-service-a

my-service-a needs to be able to communicate with those compose services (hence on the same network), e.g. consume Kafka messages.

my-service-a also exposes a REST API (which I consume from my host's browser or postman at localhost:8090). Some operations of this API cause my-service-a to perform REST calls to another app (my-service-b), which is running on my host instead, as a plain old JVM app.

my-service-a uses host name host.docker.internal in its configuration to indicate the host of my-service-b, so that it relies on name resolution and doesn't need to know a fixed IP address.

I'm quite sure that this used to work fine until very recently, so I dunno if my distro changed something, but the bottom line is, now this doesn't work anymore.

When starting my-service-a, I get this:

Error: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.docker.internal:host-gateway": host containers internal IP address is empty

So I did some searching and stumbled upon some Issues in this project (e.g. #22653). I tried to apply the first recommendation mentioned here, to no avail.

I created ~/.config/containers/containers.conf with the following (how can I be sure this change is effective?):

[network]
default_rootless_network_cmd = "pasta"
pasta_options = ["--map-gw"]

Next, I have started my-service-a again, this time without the extra host (is this ok?):

docker run -p 8090:8080 --name=my-service-a --replace --net=my-bridge_default -d org.me/my-service-a

And I have changed my-service-a's app configuration so that instead of looking for my-service-b at host.docker.internal, it uses host.containers.internal (here also, is this ok?)

Now my-service-a starts just fine, but is unable to resolve host.containers.internal, therefore it's not forwarding calls to my-service-b:

Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'host.containers.internal' [A(1)] and search domain query for configured domains failed as well: [dns.podman]

What am I doing wrong?

Thanks

P.S. I have noticed that netavark-dhcp-proxy.socket/service were not running, so I enabled those via systemd, so that they start on every boot. Not sure it's correct/needed.

podman info

Details

host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 94.41
    systemPercent: 1.31
    userPercent: 4.28
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-tumbleweed
    version: "20240823"
  eventLogger: journald
  freeLocks: 1206
  hostname: andromeda
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.10.5-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 1043873792
  memTotal: 32920047616
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.1.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.1-1.1.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun-1.15-1.1.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20240814.61c0b0d-1.1.x86_64
    version: |
      pasta 20240814.61c0b0d-1.1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 69784563712
  swapTotal: 69789020160
  uptime: 1h 24m 44.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 20
    paused: 0
    running: 7
    stopped: 13
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 707194257408
  graphRootUsed: 322045759488
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 437
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724314016
  BuiltTime: Thu Aug 22 10:06:56 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

[Luap99]

The --map-gw gw has no effect of how we resolve a ip for host.containers.internal so you would need to manually set the host entry for this with the proper gateway ip if you use this option.

If you want this entry to work automatically you will need to have a second host ip or switch the ip pasta uses via -a ... as past a options, see https://blog.podman.io/2024/03/podman-5-0-breaking-changes-in-detail/

And the latest pasta version has a new version where we can map any ip to the host so we can fix this properly in podman by v5.3 hopefully #19213, containers/common#2136

[andreaippo]

Ok thanks, I tried with those values but I think that the problem is that my pasta is not being fed that config:

❯ ps auxww | grep pasta
andrea    259592  0.0  0.0  72076 20272 ?        Ss   12:00   0:00 /usr/bin/pasta --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.0.1 -t none -u none -T none -U none --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netn

I guess ~/.config/containers/containers.conf is not the right place then, at least on my distro (opensuse tumbleweed)

EDIT: for the record, asked here

[Luap99]

The config should be the right place (unless you have XDG_CONFIG_HOME set?), but you need to stop all containers with rootless networks in order for us to stop the process so that the next start will spawn it again with the new options. If this doesn't work there could be a bug that causes us not to cleanup the root netns correctly. A reboot should be the simplest to confim it picks up the new settings afterwards.

[andreaippo]

Thanks, that env var is not set and I did stop all the containers using rootless networks, but those pasta options are not picked up nevertheless.

pasta itself is restarts as I could confirm by running ps auxww | grep pasta with all containers stopped, and there was no running process found.

So I think this is really a problem with how pasta gets its config, either specific to my distro or to this pasta release.

[andreaippo]

After a reboot today pasta took those options into account, so that part is fine now.

Will report back with the outcome of my tests

[andreaippo]

I'm speechless, it...works!

I just took the pasta config straight from here, with the exact same IP values, and put it into ~/.config/containers/containers.conf (create it if it doesn't exist):

[network]
default_rootless_network_cmd = "pasta"
pasta_options = ["-a", "10.0.2.0", "-n", "24", "-g", "10.0.2.2", "--dns-forward", "10.0.2.3"]

The default_rootless_network_cmd shouldn't be needed with Podman 5 AFAIK.

And now I have my podman container interacting with others on the same rootless bridge network, plus also able to reach a service running on my host by using host name host.containers.internal. Great!

Thanks a lot @Luap99 for your time and help - looking forward to a config-less solution with a future update, that would be cool :)