Nftables port forwarding rules leak when passing both generic (-p 8080:8080) and IPv6 wildcard (-p '[::]:8080:8080')

[danishprakash]

When using both -p 8080:8080 -p '[::]:8080:8080, netavark doesn't clean up nft chains and leaves them behind, causing problems with subsequent container runs, and port blocking.

The core issues seem to be how netavark handles port forwards when both the generic and IPv6 wildcard options are passed. In this case, duplicate dnat chains are generated for IPv6(first from the generic wildcard, and then from the IPv6 wildcard), but then netavark fails to clean up both IPv[4|6] rules, resulting in a leak. I'd expect on a container removal, all container rules are cleared. Additionally, removing the said network doesn't clear up the leaked rules either.

Reproducer (by ondrej)

podman network create test_net --ipv6
nft list table inet netavark
podman run -d \
    --name netavark-test-container \
    --rm -it \
    --network test_net \
    -p 8080:8080 \
    -p [::]:8080:8080 \
    alpine /bin/sh -c 'sleep 2'
sleep 2
nft list table inet netavark

Noticing the following errors in debug log while removing the container--container cleanup:

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

I tried deleting all the matching rules and even deduplicating them before processing with limited success. I'm still looking into a potential patch here. Ideally, either do not create the duplicate rules at all in such a situation or consider the IPv6 wildcard a no-op when a dual-stack generic wildcard port forward option is passed.

Environment

openSUSE Tumbleweed (Slowroll)
netavark latest main (and v1.16, and v1.15)
nftables v1.1.5

[Luap99]

That is a dup of #1160 but since this has a better description I am fine to keep this one open here.

For me the fix should be to not create duplicate rules in this case.

[Luap99]

But in general I would consider such port specification invalid, the reason it works today is because podman port binding logic is broken and doesn't actually handle a proper dual stack bind on no given bind address. If it would do that then such ports would fail to start as the second bind would fail with EADDRINUSE

[danishprakash]

IIRC, net.Listen("tcp"..) listens on both IPv4 and IPv6 IP addresses[1] unless or until specified otherwise via "tcp4" or "tcp6"., and podman (via rootlesskit) invokes a "tcp" listen which covers both. What podman could do here is to perhaps either ignore or error out on duplicate binds, for instance in this case the subsequent IPv6 wildcard port forward.

[1] - https://pkg.go.dev/net#Listen

[Luap99]

Yes the thing is the way podman does the binds currently is sort of broken as it always does a v4 only bind when no address was given.

https://github.com/containers/podman/blob/7fecff5c294412ab47ec7db62d8fa6af5ba0eeec/libpod/oci_util.go#L31-L138

Large parts of that must really be reworked as part of 6.0 I would say, no address must mean bind both. 0.0.0.0 should mean v4 only and :: v6 only then it would be the most logically consistent I would say.

And with that the ports would cause a bind error and the second bind as the address would already be in use.

[danishprakash]

Here's a patch that does something similar. It handles both IPv4 and IPv6 binds when no address is specified, and also prevents netavark from leaking rules. It's lenient in that if IPv6 fails to bind on the host for some reason, it will simply bind IPv4 and continue. But it errors out if IPv4 fails even in dual-stack mode, and explicit stack port binding will fail if something goes wrong, as expected.

danishprakash/podman@3f90ff3