Connect to [::1]:nodeport does not work

[uablrek]

What happened:

Access via [::1]:nodeport does not work in ipv6-only or dual-stack clusters.

What you expected to happen:

Access should work or the limitation should be documented.

How to reproduce it (as minimally and precisely as possible):

• Bring up a ipv6-only or dual-stack cluster, proxy-mode=iptables or ipvs
• Define a service of type:nodePort
• Try to access [::1]:nodeport

Anything else we need to know?:

That this does not work even for ipv4 for ipvs is reported in #67730. But this is with proxy-mode=iptables or ipvs.

The problem seem to be that there is no equivalent setting for ipv6 for;

sysctl -w net.ipv4.conf.all.route_localnet=1
route_localnet - BOOLEAN
    Do not consider loopback addresses as martian source or destination
    while routing. This enables the use of 127/8 for local routing purposes.
    default FALSE

This is set by kube-proxy (in both modes). If you set net.ipv4.conf.all.route_localnet=0 for proxy-mode=iptables access to 127.0.0.1:nodeport does not work either.

I have been told by security people that this setting opens a security issue; a host on the local segment can actually route 127.0.0.1 to a k8s-node and access ports that really should be local. Reported in #90259.

Environment:

• Kubernetes version (use kubectl version):
  v1.19.0-alpha.1.547+250884c9c1cd41
• Cloud provider or hardware configuration: kvms on Ubuntu 18
• OS (e.g: cat /etc/os-release): own setup BusyBox
• Kernel (e.g. uname -a): Linux 5.4.2 Better error messages if go isn't installed, or if gcloud is old. #2 SMP Thu Apr 16 17:58:35 CEST 2020
• Install tools: "from scratch"
• Network plugin and version (if this is a network-related bug): (any)
• Others:

[uablrek]

/sig network
/area ipv6

[uablrek]

I made a "combo" issue with 2 problems at first. Please forget that ever happened. New issue; #90259

[athenabot]

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

[uablrek]

Note; I have moved (copied) the comment below to #90259 (comment) where it belongs.

@andrewsykim Are there any reasons to set net.ipv4.conf.all.route_localnet=1 in the ipvs proxier?

If not, I will make a PR to remove it.

Consideting the problems described in #67730 I don't think is makes any difference.

[aojea]

/cc

[thockin]

Cool. This will be fun.

While it seems that localhost nodeport might be a weird thing, I know that there are people who use it. The example that immediately comes to mind is an in-cluster docker registry that uses a NodePort to be available to kubelet as "localhost:32100". This works because, while docker demands TLS for registries, it makes a special exception for localhost. So a pod can say image: localhost:32100/myimagename:tagname and it works.

Disabling localhost NodePorts would break that.

It seems that doesn't work at all on v6, which we know is not widely adopted (by comparison).

Do we:

a) normalize down: make v4 optional (default on)?
b) nomralize up: try to make localhost NodePorts work through some other mechanism?
c) other?

[uablrek]

a) If the only reason for net.ipv4.conf.all.route_localnet=1 is access 127.0.0.1:nodeport this option has my vote.

b) I have encountered this problem before when trying to get ipv6 working on K3s, k3s-io/k3s#284 (comment).
I tried quite hard to find a solution, even checked kernel code which I don't do often nowdays, but I could not find any way to make it work. But then again I missed the net.ipv4.conf.all.route_localnet for ipv4 completely so it might be a way.

And this excellent analyze; #67730 (comment) make me believe that this can not work for proxy-mode=ipvs without a kernel change.

[aojea]

I think that we should correct the title, is not totally accurate, because [::1]:nodeport works, is just that it does not work if the source address is localhost.

I've tested it creating a NodePort service, in this example listening in port 30815. I'm running all commands in the same node that runs the pod with the service.

I can access the NodePort service in the node IP address, as expected:

root@kind-worker2:/# curl [fc00:db8:2::4]:30814
<html><body><h1>It works!</h1></body></html>

but, as correctly pointed out here, I can not access it if I try to access it in the localhost address, the connection hangs, as expected:

root@kind-worker2:/# curl [::1]:30814
^C

however, if I use the node IP address to access the nodePort in localhost it works:

root@kind-worker2:/# curl --interface fc00:db8:2::4 [::1]:30814
<html><body><h1>It works!</h1></body></html>

the connection goes through because the source address "legit"

To my view route_localnet is a hack, and a) make total sense to me.

For IPv6, sooner or later we should stop to do the mapping IPv4 to IPv6 and evolve the networking model independently, leveraging the advantages of IPv6. From my POV the current kubernetes network model is heavily influenced by the IPv4 limitations regarding the limited number of addresses, being public and solved with load balancer, nodeports, NAT, ... or private, overlays networks to solve overlapping subnets problems, ...

I dream with using global addresses as ClusterIPs and move all the service plumbery to Gateway , is that too unrealistic?

[squeed]

FYI, there is no v6 equivalent to route_localnet, because it's not permitted to route packets to a larger scope.

[athenabot]

@thockin
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[athenabot]

@thockin
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[athenabot]

@thockin
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[athenabot]

@thockin
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[athenabot]

@thockin
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[aojea]

/close
ipv6 does not have the route_localnet hack , so this is working as expected
Lars reopen if you feel that we need to do anything else
Thanks

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

/close
ipv6 does not have the route_localnet hack , so this is working as expected
Lars reopen if you feel that we need to do anything else
Thanks

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.