"[Feature] KIND support for external network emulation: latency, bandwidth constraint, packet drops"

[jayunit100]

A typical kind cluster has pretty stable networking, locally:

(created running iperf against all nodes via pharb/kubernetes-iperf3)
Client on 172.18.0.2:  [  4] local 192.168.173.66 port 35378 connected to 10.107.146.212 port 5201
Client on 172.18.0.2:  [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
Client on 172.18.0.2:  [  4]   0.00-1.00   sec  1.38 GBytes  11.9 Gbits/sec  2955     
Client on 172.18.0.2:  [  4]   1.00-2.00   sec  1.02 GBytes  8.79 Gbits/sec  4071   
Client on 172.18.0.2:  [  4]   2.00-3.00   sec  1.49 GBytes  12.8 Gbits/sec  3870   

Whereas a real world cluster (for example, a high performance VMC cluster running on EC2 hardware, has a much different performance profile....

Client on 192.168.5.227:  [  4]   0.00-1.00   sec   405 MBytes  3.40 Gbits/sec  
Client on 192.168.5.227:  [  4]   1.00-2.00   sec   442 MBytes  3.71 Gbits/sec   
Client on 192.168.5.227:  [  4]   2.00-3.00   sec   461 MBytes  3.87 Gbits/sec   

It would be nice to be able to disrupt the network bandwidth and throughput on kind clusters so that they matched those of clouds . In especially congested clouds, you can even see iperf values that might be 10X less then this in peak times.... (dont have an example on hand, but if someone can run iperf in a GCE cluster with 20 parallel conformance tests running, i bet youll be able to see this).

What would you like to be added:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  disableDefaultCNI: true
  tcLatency: 100ms
  podSubnet: 192.168.0.0/16 # set to Calico's default subnet
nodes:
- role: control-plane
- role: worker
- role worker
- role: worker
- role: worker

Why is this needed:

Kind is increasingly used to simulate realistic clusters.

• In flake hunting, add hoc stuff like stress and so on are used https://kubernetes.slack.com/archives/CN0K3TE2C/p1597351797024800 commonly
• Issues with simulating GCE, EC2 and other clouds which can have high cpu or network latency spikes continue to plague upstream CI
• Developers testing locally on kind often have the benefit of extremely fast networks with very low latency. This makes cluster tests and timeout calibration for things like large database transactional workloads or http workloads with lots of traffic less realistic.

[jayunit100]

discussed with @mauilion and @BenTheElder earlier on slack:
counterpoint:

• Maullion had a clever idea , to use CNI plugins a trick for adding latency (this doesnt, however give you node and apiserver latency since those start on hostNetwork)

also

Since we want to be able to find flakes manually without relying on CI, this would enable more hypothesis driven testing that is locally runnable over time. So , likely more people would locally be able to find problematic k8s tests without relying on sporadic cloud latencys to create isolated, hard to reproduce data points.

[aojea]

I really love this idea Jay, let's discuss it further.

I think that is better to apply the traffic control "outside" of the cluster, applying the tc commands on the external veth pair interface of the nodes. KIND should continue being independent of the CNI plugins, and this way we can offer a more "real" simulation.

The implementation seems "simple", there is lot of literature on how to find the external veth interface of a container, i.e https://github.com/cslev/find_veth_docker., then, once the cluster finish the installation, to avoid causing issues with the bootstrap, we list the external interfaces of the nodes belonging to the cluster and apply the corresponding tc commands.

sudo tc qdisc add dev veth95b1019d root netem delay 200ms

this will work for any provider, docker, podman, ..

For the API I suggest to keep its own block, since the traffic control allow to control another interesting parameters like bandwidth and traffic drops, something like:

networking:
  netem:
    delay: 100ms
    bandwidth:
       rate: 1M
       burst: 25k
    loss: 10%

[aojea]

/retitle "[Feature] KIND support for external network emulation: latency, bandwidth constraint, packet drops"

[aojea]

/assign @BenTheElder
for approval of the feature request

[jayunit100]

Yayyyyyy

[aojea]

I've been playing with this and it can be done simply with a bash scrip, please take a look.

https://gist.github.com/aojea/603e88ba709aac874bd7611752261772

I think that is better to get feedback and then, if there is demand, include it in KIND,

[jayunit100]

thanks @aojea ! How would you suggest publishing and standardizing this script, if we dont do it in Kind? I guess we could put it on a personal github repo as a kind recipe of some sort (i have many of my own https://github.com/jayunit100/k8sprototypes/tree/master/kind ) maybe we could join forces or something.

,.. i think if it goes in kind as an option, we'll get alot of great opportunities for newcomers to help simulate CI on their laptops , so it would be a potential big boost to the test-infra folks long term if we build some community around it.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[bingzheliu]

Hello, I am planning to integrate kind with a network emulator, Mininet, in particular. Thanks for @aojea to implement traffic control. But I may need more than this: I would want to setup the network topology like in a datacenter (e.g. fattree topology), and run various network routing protocol.

I've got some basic understanding on how Kind implement, and I have two plans to do the implementation. I am wondering if you guys could help me to decide between the two.

The basic goal is to replace kind's user-bridged networking with Mininet (actually I use Containernet, where I've modified it to be able to connect running containers, by placing a veth into the running container namespace). Now I am working on the Kind side. Two plans are:

1. Not modifying Kind, let it finishes, and then delete the network interfaces, modify all the related Kubernete configuration (though I've not fully figured out which files I need to modify, any hints will be very appreciated).
2. Modify kind: (a) adding a few args to indicate setting up with Mininet (b) start container without any networking ("none") by setting arg at this line, (c) before setting up k8s, trigger Mininet to run and setup the network. Then run the kubeadm init and so on.

Any feedback would be very appreciated!

[aojea]

• Not modifying Kind, let it finishes, and then delete the network interfaces, modify all the related Kubernete configuration (though I've not fully figured out which files I need to modify, any hints will be very appreciated).
• Modify kind: (a) adding a few args to indicate setting up with Mininet (b) start container without any networking ("none") by setting arg at this line, (c) before setting up k8s, trigger Mininet to run and setup the network. Then run the kubeadm init and so on.

You have more details in this link about how to setup complex scenarios with KIND https://gist.github.com/aojea/00bca6390f5f67c0a30db6acacf3ea91

I suggest you to start with 1), start small and iterate, you can always move from 1) to 2) later ... I don't discard that as soon as you start to be more familiar with the environment and the problematic you'll have new options ;)

[bingzheliu]

Thank you @aojea !! For 1), could you please provide some hints on which files I need to modify? I found things like this discussion, does this look complete? Also found here saying that the certificate is signed for the old IP address... Do I need to redo the cert generation? ...

[aojea]

You have a detailed description of the modifications needed here
https://gist.github.com/aojea/00bca6390f5f67c0a30db6acacf3ea91#example-multiple-network-interfaces-and-multi-home-nodes

[bingzheliu]

@aojea I followed the steps to modify the IP. I am struggling with one step: "When creating the cluster we must add the loopback IP address of the control plane to the certificate SAN (the apiserver binds to "all-interfaces" by default)". Could you please advise how to modify Kind's configuration to modify the certificate SAN (I can't quite find which func in Kind is related to the certificate SAN)? Thank you!

[aojea]

it has to be patched in kubeadm, in the kind config, replace my-loopback with the apiserver loopback address

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# add the loopback to apiServer cert SANS
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
  kind: ClusterConfiguration
  patch: |
    - op: add
      path: /apiServer/certSANs/-
      value: my-loopback

I really don't know if is possible to modify it after the installation and how to do it

[bingzheliu]

Thank you @aojea. The following somehow did not work for me. So I finally chose "plan 3", to just delete the eth0 on kind container, and put veth into the namespace assigning the same IP address. It looks work now :)

it has to be patched in kubeadm, in the kind config, replace my-loopback with the apiserver loopback address

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# add the loopback to apiServer cert SANS
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
  kind: ClusterConfiguration
  patch: |
    - op: add
      path: /apiServer/certSANs/-
      value: my-loopback

I really don't know if is possible to modify it after the installation and how to do it

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

I will let @aojea decide if we try to wire this into kindnetd etc or leave it to external extension.

[aojea]

I demoed this in the kubecon 2021 EU and created a plugin using kind API
Source code and presentation are available in https://github.com/aojea/kind-networking-plugins

If there is more traction we can consider to move the feature to the core, but I don't have the feeling this should be part of it right now

/close

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

I demoed this in the kubecon 2021 EU and created a plugin using kind API
Source code and presentation are available in https://github.com/aojea/kind-networking-plugins

If there is more traction we can consider to move the feature to the core, but I don't have the feeling this should be part of it right now

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.