-
Notifications
You must be signed in to change notification settings - Fork 10
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adds support for MultiNetworkPolicy definitions (#907)
* Adds support for MultiNetworkPolicy objects We want to be able to apply NetworkPolicy definitions to experiment pods, but your CNI plugin needs to support applying NetworkPolicy definitions. We use both flannel and multus, either of which support this. However, there is another effort from the k8s Network Plumbing Working Group (same people who maintain multus) to implement a system that allows NetworkPolicy definitions to pods with multiple interfaces. https://github.com/k8snetworkplumbingwg/multi-networkpolicy https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables * Condenses multi-networkpolicy configmaps into a single one * Fixes location of closing bracket in multi-networkpolicy configmap * Fixes multi-networkplicy DaemonSet and add test policy for NDT * Changes experiment net-attach-defs to CNI v0.3.1 It was previously v0.2.0, which is what index2ip was designed around, but I discovered that using that version was impacting multus' functionality with regard to annotating pods correctly, and possibly in other ways we weren't even noticing. * Use v1beta2 of MultiNetworkPolicy This version introduces the "endPort" field, allowing you to specify a port range, which we need to do for ndt5. Also, allow additional ports for ndt: 3001, 3010, and ephemeral port range. The range reflects the value of net.ipv4.ip_local_port_range on our Ubuntu systems. * Removes namespaceSelector from MultiNetworkPolicy It was not valid in that location. * Uses image from m-lab fork of multi-networkpolicy-iptables Also updates ICMP flags to use "accept" instead of "allow" * Adds MultiNetworkPolicy defs for msak, revt, wehe and neubot * Modifies the allowed ports for revtr in its MultiNetworkPolicy * Loads MultiNetworkPolicies for mask, revtr, wehe and neubot * Adds podSelectors for MultiNetworkPolicies After adding multiple MultiNetworkPolicies the multi-networkpolicy DaemonSets seems to not be applying the rules properly. I had figured that the NetworkAttachmentDefinition would be sufficient. This may or may not change/fix anything. * Formats MultiNetworkPolicies as jsonnet I had accidentally inserted some plain JSON. * Removes podSelector from MultiNetworkPolicies It turns out the issue was apparently just me pushing changes to k8s-support in sandbox with the multi-networkpolicy changes, and then another branch without, and then with, and then without, which got pod annotations all messed up. * Removes sample iptables custom rules from configmap The rules are not utilized by multi-networkpolicy unless you pass special flags, which are outlined in a comment in the file. * Adds new MultiNetworkPolicy template to templates.jsonnet * Adds a MultiNetworkPolicy definition to the ndt.jsonnet manifest * Adds MultiNetworkPolicy definition to revtr manifest * Adds MultiNetworkPolicy definition to the wehe manifest * Adds MultiNetworkPolicy definition to msak manifest * Adds MultiNetworkPolicy definition to neubot manifest * Removes static MultiNetworkPolicy definitions These are now handled by a new template, and called from each experiment manifest. * Flattens arrays for experiment imports in system.jsonnet The imports for experiments are now an array consisting of the experiment DaemonSet definition and it's corresponsing MultiNetworkPolicy definition. * Removes import of now nonexistent static MultiNetworkPolicies * Casts endPort string to an int The MultiNetworkPolicy CRD schema doesn't allow for that field to be a string, unlike "port", which is odd, but whatever. * Adds additional comments clarify custom iptables rules
- Loading branch information
Showing
12 changed files
with
1,624 additions
and
513 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| { | ||
| kind: 'ConfigMap', | ||
| apiVersion: 'v1', | ||
| metadata: { | ||
| name: 'multi-networkpolicy-custom-rules', | ||
| namespace: 'kube-system', | ||
| labels: { | ||
| tier: 'node', | ||
| app: 'multi-networkpolicy', | ||
| }, | ||
| }, | ||
| // Add custom iptables rules below. The rules will not be applied unless you | ||
| // pass at least one of the following flags to multi-networkpolicy-iptables | ||
| // in the multi-networkpolicy DaemonSet in | ||
| // k8s/daemonsets/core/multi-networkpolicy.jsonnet: | ||
| // | ||
| // --custom-v4-igress-rule-file | ||
| // --custom-v4-egress-rule-file | ||
| // --custom-v6-igress-rule-file | ||
| // --custom-v4-egress-rule-file | ||
| // | ||
| // Add iptables rules one per line in the appropriate sections below, minus | ||
| // "iptables -A <chain>" as that is added for you by | ||
| // multi-networkpolicy-iptables. | ||
| data: { | ||
| 'custom-v4-ingress-rules.txt': ||| | ||
| # No custom rules, yet. | ||
| |||, | ||
| 'custom-v4-egress-rules.txt': ||| | ||
| # No custom rules, yet. | ||
| |||, | ||
| 'custom-v6-ingress-rules.txt': ||| | ||
| # No custom rules, yet. | ||
| |||, | ||
| 'custom-v6-egress-rules.txt': ||| | ||
| # No custom rules, yet. | ||
| |||, | ||
| }, | ||
| } | ||
|
|
775 changes: 775 additions & 0 deletions
775
k8s/custom-resource-definitions/multi-networkpolicy.jsonnet
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| { | ||
| apiVersion: 'apps/v1', | ||
| kind: 'DaemonSet', | ||
| metadata: { | ||
| name: 'multi-networkpolicy', | ||
| namespace: 'kube-system', | ||
| labels: { | ||
| tier: 'node', | ||
| app: 'multi-networkpolicy', | ||
| name: 'multi-networkpolicy', | ||
| }, | ||
| }, | ||
| spec: { | ||
| selector: { | ||
| matchLabels: { | ||
| name: 'multi-networkpolicy', | ||
| }, | ||
| }, | ||
| updateStrategy: { | ||
| type: 'RollingUpdate', | ||
| }, | ||
| template: { | ||
| metadata: { | ||
| labels: { | ||
| tier: 'node', | ||
| app: 'multi-networkpolicy', | ||
| name: 'multi-networkpolicy', | ||
| }, | ||
| }, | ||
| spec: { | ||
| hostNetwork: true, | ||
| nodeSelector: { | ||
| 'kubernetes.io/arch': 'amd64', | ||
| }, | ||
| tolerations: [ | ||
| { | ||
| operator: 'Exists', | ||
| effect: 'NoSchedule', | ||
| }, | ||
| ], | ||
| serviceAccountName: 'multi-networkpolicy', | ||
| containers: [ | ||
| { | ||
| name: 'multi-networkpolicy', | ||
| image: 'measurementlab/multi-networkpolicy-iptables:latest', | ||
| imagePullPolicy: 'Always', | ||
| command: [ | ||
| '/usr/bin/multi-networkpolicy-iptables', | ||
| ], | ||
| args: [ | ||
| '--accept-icmp', | ||
| '--accept-icmpv6', | ||
| '--container-runtime-endpoint=/run/containerd/containerd.sock', | ||
| '--host-prefix=/host', | ||
| '--network-plugins=netctl,ipvlan', | ||
| '--pod-iptables=/var/lib/multi-networkpolicy/iptables', | ||
| // If any custom iptables rules are needed that cannot be | ||
| // provisioned by MultiNetworkPolicy definitions, then you can | ||
| // add them to the file configs/multi-networkpolicy.jsonnet in | ||
| // this repo, and uncomment the following flags as necessary: | ||
| // '--custom-v4-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-ingress-rules.txt', | ||
| // '--custom-v6-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-ingress-rules.txt', | ||
| // '--custom-v4-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-egress-rules.txt', | ||
| // '--custom-v6-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-egress-rules.txt', | ||
| ], | ||
| resources: { | ||
| requests: { | ||
| cpu: '100m', | ||
| memory: '80Mi', | ||
| }, | ||
| limits: { | ||
| cpu: '100m', | ||
| memory: '150Mi', | ||
| }, | ||
| }, | ||
| securityContext: { | ||
| privileged: true, | ||
| capabilities: { | ||
| add: [ | ||
| 'SYS_ADMIN', | ||
| 'NET_ADMIN', | ||
| ], | ||
| }, | ||
| }, | ||
| volumeMounts: [ | ||
| { | ||
| name: 'host', | ||
| mountPath: '/host', | ||
| }, | ||
| { | ||
| name: 'var-lib-multinetworkpolicy', | ||
| mountPath: '/var/lib/multi-networkpolicy', | ||
| }, | ||
| { | ||
| name: 'multi-networkpolicy-custom-rules', | ||
| mountPath: '/etc/multi-networkpolicy/rules', | ||
| readOnly: true, | ||
| }, | ||
| ], | ||
| }, | ||
| ], | ||
| volumes: [ | ||
| { | ||
| name: 'host', | ||
| hostPath: { | ||
| path: '/', | ||
| }, | ||
| }, | ||
| { | ||
| name: 'var-lib-multinetworkpolicy', | ||
| hostPath: { | ||
| path: '/var/lib/multi-networkpolicy', | ||
| }, | ||
| }, | ||
| { | ||
| name: 'multi-networkpolicy-custom-rules', | ||
| configMap: { | ||
| name: 'multi-networkpolicy-custom-rules', | ||
| }, | ||
| }, | ||
| ], | ||
| }, | ||
| }, | ||
| }, | ||
| } | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.