Rootless Nginx Wrong "remote_addr" while Rootful Nginx has correct "remote_addr"

[usury]

Issue Description

For rootless nginx containers, $remote_addr takes the internal address of the container as evidenced in the container logs.
For rootful nginx containers, $remote_addr is the correct ip address of the requesting system, as one would expect.

This is a problem when using a rootless nginx container as a reverse proxy for other rootless webserver containers (the use case where I encountered the problem).

• if reverse proxy nginx container never receives correct $remote_addr it can never accurately set HTTP_X_FORWARDED_FOR or HTTP_X_REAL_IP to propagate to upstream webserver
• upstream webserver, therefore, can never know the correct ip address of the requesting system

Steps to reproduce the issue

Steps to reproduce the issue

Rootless Example
All podman actions performed as a normal user on containerhost
This example also shows unexpected (though not necessarily related) blank "IPAddress"

normaluser@containerhost $> podman rm -fa       # fresh set of examples
normaluser@containerhost $> podman run --name nginx8080.rootless --publish 8080:80 --rm -d nginx:alpine
normaluser@containerhost $> podman inspect nginx8080.rootless | grep -i ipaddress
          "IPAddress": "",
          # this is unexpected - I don't know why this is blank - or if it's related
normaluser@containerhost $> podman exec nginx8080.rootless ping -c 1 nginx8080.rootless
          PING nginx8080.rootless (10.0.2.100): 56 data bytes
          64 bytes from 10.0.2.100: seq=0 ttl=42 time=0.033 ms
          # container can ping itself by name
normaluser@containerhost $> podman exec nginx8080.rootless cat /etc/hosts | grep 10.0.2.100
          10.0.2.100	a4ed5a393e55 nginx8080.rootless
          # container's "/etc/hosts" contains an IP

someuser@OTHERHOST $> nslookup OTHERHOST
          Address: 192.168.10.12
someuser@OTHERHOST $> curl http://containerhost:8080
          # correct nginx generic landing page

normaluser@containerhost $> podman logs nginx8080.rootless
          10.0.2.100 - - [13/Mar/2023:12:56:10 -0700] "GET / HTTP/1.1" 200 615 "-" "curl/7.79.1" "-"
          # ^--- $remote_addr is CONTAINER's IP Addr, not OTHERHOST's IP Addr (and not CONTAINERHOST IP, either)

Rootful Example
All podman actions performed as superuser on containerhost

root@containerhost:~ # podman rm -fa
root@containerhost:~ # podman run --name nginx8081.rootful --publish 8081:80 --rm -d nginx:alpine
root@containerhost:~ # podman inspect nginx8081.rootful | grep -i ipaddress
          "IPAddress": "10.88.0.4"
          # note, this container can also ping itself by name and has proper container /etc/hosts entry

someuser@OTHERHOST $> nslookup OTHERHOST
          Address: 192.168.10.12
someuser@OTHERHOST $> curl http://containerhost:8081
          # correct nginx generic landing page

root@containerhost:~ # podman logs nginx8081.rootful
          192.168.10.12 - - [13/Mar/2023:20:14:21 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/7.79.1" "-"
          # ^--- $remote_addr is correct OTHERHOST's IP Addr

Describe the results you received

Incorrect $remote_addr in nginx logs for rootless container instances.
This also means nginx as a reverse proxy cannot be configured to pass correct $remote_addr via HTTP_X_FORWARDED_FOR or HTTP_X_REAL_IP headers.

Describe the results you expected

Correct $remote_addr in nginx logs for both rootless and rootful container instances.

podman info output

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.2, commit: '
  cpus: 1
  distribution:
    codename: bullseye
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: arachne
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.0-21-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 531066880
  memTotal: 1024032768
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/2000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 0
  swapTotal: 0
  uptime: 13h 50m 19.11s (Approximately 0.54 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/XXXXXXX/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/XXXXXXX/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 8
  runRoot: /run/user/2000/containers
  volumePath: /home/XXXXXXX/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.15
  OsArch: linux/amd64
  Version: 3.4.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

I'm running podman in a Debian 11 x86_64 VM
Debian does not maintain a recent podman in its repos (v3.1.x iirc)
I'm using the most recent packages available to me from the OpenSuse "kubic" repo

podman --version
podman version 3.4.2

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[usury]

I did a little more digging and got the trivial example I posted to work properly. It seems related to functionality introduced in PR 6965 (#6965) for a bug related to "only 127.0.0.1 within containers", also having to do with throughput optimizations.

A solution to the trivial example appears here. However, my actual use case involves a user-defined rootless pod with a reverse proxy and multiple web servers all on a user-defined rootless CNI network. The solution appearing here does not work in my use case. I will devise a simplified example and post an new bug report.

The change is --network slirp4netns:port_handler=slirp4netns

normaluser@containerhost $> podman rm -fa       # fresh set of examples
normaluser@containerhost $> podman run --name myNginx8081.rootless --publish 8081:80 \
                                       --network slirp4netns:port_handler=slirp4netns \
                                       --rm -d nginx:alpine
normaluser@containerhost $> podman run --name myNginx8082.rootless --publish 8082:80 \
                                       --network slirp4netns:port_handler=slirp4netns \
                                       --rm -d nginx:alpine

someuser@OTHERHOST $> curl http://containerhost:8081

normaluser@containerhost $> podman logs myNginx8081.rootless
          192.168.10.12 - - [13/Mar/2023:14:12:24 -0700] "GET / HTTP/1.1" 200 615 "-" "curl/7.79.1" "-"
          # ^--- $remote_addr is correct OTHERHOST's IP Addr

The "no ip address using podman inspect" still exists though.
Shall I open a separate bug report?
I don't know the impact of not having an ip address appear there.

normaluser@containerhost $> podman inspect myNginx8081.rootless | grep -i ipaddress
          "IPAddress": "",

The container can still ping itself, and the container's /etc/hosts is correct, same as above in the rootless example.
However, each container has the same IP address.
This may be perfectly sensible and working as designed since these containers aren't on a user-defined network with each other.

normaluser@containerhost $> podman exec myNginx8081.rootless cat /etc/hosts
          10.0.2.100	a00b2b1321d2 myNginx8081.rootless
normaluser@containerhost $> podman exec myNginx8082.rootless cat /etc/hosts
          10.0.2.100	6aec236dcf5b myNginx8082.rootless

When using port_handler=slirp4netns, or maybe when containers aren't on a user-defined network with each other, it means containers cannot reach each other by name (only by IP addr, which doesn't appear when using podman inspect on the container).
This is a problem for reverse proxy setups expecting to reach containers by name (my use case).

normaluser@containerhost $>  podman exec myNginx8081.rootless ping myNginx8082.rootless
        ping: bad address 'myNginx8082.rootless'
normaluser@containerhost $>  podman exec myNginx8082.rootless ping myNginx8081.rootless
        ping: bad address 'myNginx8081.rootless'

[sbrivio-rh]

Debian does not maintain a recent podman in its repos (v3.1.x iirc)

Not really, you can even use the latest minor version, 4.4, from experimental.

The change is --network slirp4netns:port_handler=slirp4netns

That's also documented in the man page by the way:

Note: Rootlesskit changes the source IP address of incoming packets to an IP address in the container
network namespace, usually 10.0.2.100. If your application requires the real source IP address, e.g.
web server logs, use the slirp4netns port handler

The "no ip address using podman inspect" still exists though. Shall I open a separate bug report?

Probably, yes.

You could also try out the new network mode --network pasta, which preserves the correct source IP address on forwarded packets.

[Luap99]

As @sbrivio-rh mentioned this is documented, for user defined networks it will not work, #8193.

IMO it makes no sense to show an ip address in inspect there, this will just add confusion because the ip address not routable from anywhere beside the container netns. Each container runs their own slirp4netns instance so they cannot talk to each other in this case, you need to use forwarded host ports for that.