Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hostPort conflicts with IPVS on 1.7.x #9223

Open
Max-Sum opened this issue Aug 25, 2024 · 1 comment
Open

hostPort conflicts with IPVS on 1.7.x #9223

Max-Sum opened this issue Aug 25, 2024 · 1 comment

Comments

@Max-Sum
Copy link

Max-Sum commented Aug 25, 2024

Bug Report

After upgrading to 1.7.6. If a pod listen to hostPort, any request to services with the same port number on the same node will be redirected to this hostPort.

Description

I have a traefik pod listen to hostPort 80, 443 that masks requests to kubernetes API 10.96.0.1:443.

Logs

After 1.7.x, some iptables rules use iptables-nft while hostPort-related items continue to use iptables-legacy.

On 1.6.8:

$ iptables-legacy-save
# Generated by iptables-save v1.8.9 on Sun Aug 25 18:08:14 2024
*nat
:PREROUTING ACCEPT [99:12216]
:INPUT ACCEPT [50:7761]
:OUTPUT ACCEPT [44:7576]
:POSTROUTING ACCEPT [86:11523]
:CNI-DN-b618f359d5a1c38a30eef - [0:0]
:CNI-DN-f7d536801c41904670fe8 - [0:0]
:CNI-DN-f94cfd0b249b096d6a37a - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.9.0/24 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A CNI-DN-b618f359d5a1c38a30eef -s 10.244.9.0/24 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-b618f359d5a1c38a30eef -s 127.0.0.1/32 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-b618f359d5a1c38a30eef -p udp -m udp --dport 3478 -j DNAT --to-destination 10.244.9.67:3478
-A CNI-DN-f7d536801c41904670fe8 -s 10.244.9.0/24 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -s 127.0.0.1/32 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.244.9.63:8000
-A CNI-DN-f7d536801c41904670fe8 -s 10.244.9.0/24 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -s 127.0.0.1/32 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.244.9.63:8443
-A CNI-DN-f7d536801c41904670fe8 -s 10.244.9.0/24 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -s 127.0.0.1/32 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -p tcp -m tcp --dport 4430 -j DNAT --to-destination 10.244.9.63:4430
-A CNI-DN-f94cfd0b249b096d6a37a -s 10.244.9.0/24 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f94cfd0b249b096d6a37a -s 127.0.0.1/32 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f94cfd0b249b096d6a37a -p udp -m udp --dport 8443 -j DNAT --to-destination 10.244.9.66:8443
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cbr0\" id: \"a2b80f7a9e40d142b740cd6a6c2a812c98c963ac1ba72c49e3c58bc872fcdd3c\"" -m multiport --dports 80,443,4430 -j CNI-DN-f7d536801c41904670fe8
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"85ccb4c2d18139b2505184edca9f9fdd3ba8a07b48e7a6c4ae5d5ee6689d7f07\"" -m multiport --dports 8443 -j CNI-DN-f94cfd0b249b096d6a37a
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"1f633b9873e3ec61f505732bf872b6dfa45b448997c70d838553918b2dda67b9\"" -m multiport --dports 3478 -j CNI-DN-b618f359d5a1c38a30eef
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.244.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
COMMIT
# Completed on Sun Aug 25 18:08:14 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 18:08:14 2024
*filter
:INPUT ACCEPT [796:154156]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [752:67420]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-PROXY-FIREWALL - [0:0]
:KUBE-SOURCE-RANGES-FIREWALL - [0:0]
-A INPUT -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A INPUT -m comment --comment "kubernetes health check rules" -j KUBE-NODE-PORT
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-NODE-PORT -m comment --comment "Kubernetes health check node port" -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j ACCEPT
-A KUBE-SOURCE-RANGES-FIREWALL -j DROP
COMMIT
# Completed on Sun Aug 25 18:08:14 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 18:08:14 2024

On 1.7.6:

$ iptables-legacy-save
# Generated by iptables-save v1.8.9 on Sun Aug 25 17:49:54 2024
*nat
:PREROUTING ACCEPT [1740:192951]
:INPUT ACCEPT [1040:122485]
:OUTPUT ACCEPT [1674:133000]
:POSTROUTING ACCEPT [2176:183720]
:CNI-DN-3fbb375130723d1a84cbe - [0:0]
:CNI-DN-81077802a72ed5ac760e9 - [0:0]
:CNI-DN-83011c2a8020d81033ef2 - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A CNI-DN-3fbb375130723d1a84cbe -s 10.244.9.0/24 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3fbb375130723d1a84cbe -s 127.0.0.1/32 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3fbb375130723d1a84cbe -p udp -m udp --dport 8443 -j DNAT --to-destination 10.244.9.58:8443
-A CNI-DN-81077802a72ed5ac760e9 -s 10.244.9.0/24 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -s 127.0.0.1/32 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.244.9.60:8000
-A CNI-DN-81077802a72ed5ac760e9 -s 10.244.9.0/24 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -s 127.0.0.1/32 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.244.9.60:8443
-A CNI-DN-81077802a72ed5ac760e9 -s 10.244.9.0/24 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -s 127.0.0.1/32 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -p tcp -m tcp --dport 4430 -j DNAT --to-destination 10.244.9.60:4430
-A CNI-DN-83011c2a8020d81033ef2 -s 10.244.9.0/24 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-83011c2a8020d81033ef2 -s 127.0.0.1/32 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-83011c2a8020d81033ef2 -p udp -m udp --dport 3478 -j DNAT --to-destination 10.244.9.59:3478
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"f4cfae4c920976730e9649b9085239b58ceb374d0de72f6aa90e8678323fb437\"" -m multiport --dports 8443 -j CNI-DN-3fbb375130723d1a84cbe
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"1bb40bd499077e221a51b01c61910c2dfd8a2f4c7e40f28c574faeb2e90f6579\"" -m multiport --dports 3478 -j CNI-DN-83011c2a8020d81033ef2
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cbr0\" id: \"10d281defe95b891415a9f5da51109459626ab584b58827822834b6959f425d2\"" -m multiport --dports 80,443,4430 -j CNI-DN-81077802a72ed5ac760e9
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
COMMIT
# Completed on Sun Aug 25 17:49:54 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 17:49:54 2024
*filter
:INPUT ACCEPT [18325:6493414]
:FORWARD ACCEPT [68118:14066587]
:OUTPUT ACCEPT [18124:2101431]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
-A INPUT -j KUBE-FIREWALL
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
# Completed on Sun Aug 25 17:49:54 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 17:49:54 2024
*mangle
:PREROUTING ACCEPT [84803:20325123]
:INPUT ACCEPT [18333:6495024]
:FORWARD ACCEPT [68118:14066587]
:OUTPUT ACCEPT [18662:2137631]
:POSTROUTING ACCEPT [86250:16169628]
:KUBE-IPTABLES-HINT - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
COMMIT
# Completed on Sun Aug 25 17:49:54 2024


$ iptables-save
# Generated by iptables-save v1.8.9 (nf_tables) on Sun Aug 25 17:49:11 2024
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-IPTABLES-HINT - [0:0]
COMMIT
# Completed on Sun Aug 25 17:49:11 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Sun Aug 25 17:49:11 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-PROXY-FIREWALL - [0:0]
:KUBE-SOURCE-RANGES-FIREWALL - [0:0]
-A INPUT -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A INPUT -m comment --comment "kubernetes health check rules" -j KUBE-NODE-PORT
-A FORWARD -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-NODE-PORT -m comment --comment "Kubernetes health check node port" -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j ACCEPT
-A KUBE-SOURCE-RANGES-FIREWALL -j DROP
COMMIT
# Completed on Sun Aug 25 17:49:11 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Sun Aug 25 17:49:11 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.9.0/24 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.244.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
COMMIT
# Completed on Sun Aug 25 17:49:11 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

Environment

  • Talos version: v1.7.6
  • Kubernetes version: 1.25.2
  • Platform: Oracle
@smira
Copy link
Member

smira commented Aug 26, 2024

Have you upgraded your Kubernetes manifests after an upgrade?

Look what's in -legacy and try to see which component is still using it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@smira @Max-Sum