Network policy enforcement is delayed #947
Milestone
Comments
|
@williamsandrew thanks for reporting this! I ran into this as well. I figure it would be useful to be able to reproduce this for anyone that knows enough to try resolve this, and here is how. My environmentUbuntu 20.04, k3s Reproduction scriptToggle meecho "Installing k3s..."
curl -sfL https://get.k3s.io | sh -s - \
--write-kubeconfig-mode=644 \
--disable metrics-server \
--disable traefik \
--disable local-storage \
--docker
echo "Installing a networkpolicy, service, and deployment for an echoserver ..."
cat << EOF | k3s kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: echoserver
spec:
podSelector:
matchLabels:
app: echoserver
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
echoserver-access: "true"
---
apiVersion: v1
kind: Service
metadata:
name: echoserver
spec:
ports:
- port: 80
targetPort: 8080
selector:
app: echoserver
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echoserver
spec:
selector:
matchLabels:
app: echoserver
template:
metadata:
labels:
app: echoserver
spec:
containers:
- name: echoserver
image: gcr.io/google_containers/echoserver:1.10
ports:
- containerPort: 8080
EOF
echo "waiting for the deployed echoserver to be ready ..."
k3s kubectl rollout status --watch deploy/echoserver
echo "Trying to access echoserver without the correct label for access..."
k3s kubectl run -it --rm --restart=Never --image=busybox --generator=run-pod/v1 busybox-without-access -- sh -c 'while ! wget -q -O- http://$ECHOSERVER_SERVICE_HOST; do sleep 5; done'
echo "Waiting two minutes to see if something changes..."
sleep 120
echo "Trying to access echoserver with the correct label for access..."
k3s kubectl run -it --rm --restart=Never --image=busybox --generator=run-pod/v1 --labels=echoserver-access=true busybox-with-access -- sh -c 'while ! wget -q -O- http://$ECHOSERVER_SERVICE_HOST; do sleep 5; done'ObservationThe busybox-without-access had access, and then after waitin for two minutes, the busybox-with-access failed to get access during the initial ~40 seconds, and then got access. I see two issues which may relate:
DiagnosticsOutput of: journalctl -u k3s --since "10 minutes ago"Output of: k3s kubectl get eventsOutput of: k3s kubectl logs deploy/echoserver |
Closed
Logs from using containerd (no
|
|
We are using a an implementation called kube-router (https://github.com/cloudnativelabs/kube-router/blob/856c7d762a73df557e0a1d35721f48fe8ba7925d/pkg/controllers/netpol/network_policy_controller.go#L58) which includes a syncPeriod as part of it's operation, please feel free to open an issue there if it makes sense. |
|
Thanks. |
9 tasks
|
I just realized (even though it's mentioned in the kube-router issue I linked above) that this does not appear to be an issue with every network policy rule. |
|
Reproduced the issue in k3s version v1.17.4+k3s1 Validated on k3s v1.20.0-rc5+k3s1, there is no delay in policy enforcement. |
|
It looks like there is a If I'm understanding the description correctly, it sounds like changing this value could lower the amount of time it takes kube-router to enforce NetworkPolicy changes. Is it possible to make this option configurable in k3s? |
|
Not at the moment. With recent updates to the network policy controller, the out of the box settings should be sufficient. |
|
Thanks Brandon, I didn't catch the end of Shylaja's message above. I verified that there is no NetworkPolicy delay on k3d v4.3.0 with k3s v1.20.4-k3s1 👍 |
|
It seems like this is still happening on recent versions.
|
|
This is a limitation of network policies. Sync of policy rules is not instantaneous and does not block pod startup. Kubernetes is build around the concept of eventual consistency. Any component that has a dependency on the state of another component should be able to retry as necessary until that component becomes available - or reachable in this case. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Version:
k3s version v0.10.0 (f9888ca3)Describe the bug
There appears to be period of time (<1 minute) after a pod is started during which applicable network policies are not enforced.
To Reproduce
Steps to reproduce the behavior:
https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/11-deny-egress-traffic-from-an-application.md
Expected behavior
All egress requests (via
wget) from theapp=foolabeled pod should fail with eitherbad addressordownload timed out(depending on the network policy in use).Actual behavior
Egress requests (via
wget) from theapp=foolabeled pod initially succeed after pod startup before eventually failing as expected.Additional context
The text was updated successfully, but these errors were encountered: