Standardize k8s networking policies (CNI)

[garloff]

As SCS Managed Kubernetes Operator, I want to understand precisely which k8s networking policies I need to implement and which conformance tests I need to pass for compliance.
As SCS user, I want to understand what networking policy support I can rely on when automating my container workload deployment/management.

(This belongs to Epic #615.)

This story involves:

• Research: Understanding the precise standards. Are they comprehensive and clear or do we need to amend them?
• Research: Do these policies cover load-balancers? If so, is this something we can implement?
• Research: What (conformance) tests do exist that we can leverage?
• Seeking input/alignment from upstream communities / SCS friends and family as needed.
• Write down the standard (in SCS standards form, see ADR-0001) - prefer referencing existing standards, of course
• Implement conformance tests.
• Ensure our reference implementation passes.

Definition of Ready:

• User Story is small enough to be finished within one sprint
• User Story is clear and understood by the whole team
• Acceptance criteria are defined
• Acceptance criteria are clear and understood by the whole team

Definition of Done:

• All acceptance criteria are met
• Changes have been reviewed
• CI tests have run successfully
• Documentation has been updated
• Release Notes have been updated

[garloff]

Kubernetes E2E tests do cover at least some of this.

[NotTheEvilOne]

Findings

• We have focused / settled on Cilium [1] in the mean time, so particular restrictions / features apply. This is true (v1.14) for some rather basic features [2] including but not limited to ipBlock set with a pod IP and Port ranges (endPort).
• Policies do not apply to LoadBalancers, as definitions focus on namespaces and pods [3] only. Quoting k8s.io: In the case of ingress, this means that in some cases you may be able to filter incoming packets based on the actual original source IP, while in other cases, the "source IP" that the NetworkPolicy acts on may be the IP of a LoadBalancer or of the Pod's node, etc. That means that at the time of writing all pods must be selected for rules to be applied [4]. Quoting a matching request at stack overflow: [...] Then I created this network policy to make sure other pods in the cluster won't be able to connect to it anymore [...] However, it surprised me that using my external browser I also can't connect anymore to it through the load balancer [...] If I delete the policy it starts to work again. [...].

Conclusion
With only one port being available to set rules for and the need to allow all pods to accept incoming traffic at the given port for Cilium v1.14 it seems to be complicated to standardize rules with significant and useful impact. However the limitation of not supporting port ranges may change at v1.15. It should be reconsidered therefore at a future release.

[1] SovereignCloudStack/k8s-cluster-api-provider#431
[2] https://docs.cilium.io/en/v1.14/network/kubernetes/policy/#networkpolicy-state
[3] https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors
[4] https://stackoverflow.com/questions/47327554/kubernetes-networkpolicy-allow-loadbalancer