k8s-bench-suite

Bash scripts collection to benchmark kubernetes cluster performance

knb : Kubernetes Network Benchmark

knb is a bash script that will start a networking benchmark on a target Kubernetes cluster.

Here are some highlights:

• Plain bash script with very few dependencies
• Complete benchmark takes only 2 minutes
• Ability to select only a subset of benchmark tests to run
• Testing both TCP and UDP bandwidth
• Automatic detection of CNI MTU
• Includes host cpu and ram monitoring in benchmark report
• Ability to create static graph images based on the result data using plotly/orca (see examples below)
• No ssh access required, just an access to the target cluster through standard kubectl
• No need for high privileges, the script will just launch very lightweight pods on two nodes.
• Based on very lights containers images :
  • infrabuilder/bench-iperf3, is used to run benchmark tests
  • infrabuilder/bench-custom-monitor, is used to monitor nodes
• Ability to run the whole suite in a container olegeech/k8s-bench-suite:
  • Image is based on the bitnami/kubectl
  • Nodes for testing can be auto-preselected

Requirements

This script needs a valid kubectl setup with an access to the target cluster.

Binaries dependencies for the host that will execute knb :

• awk
• grep
• tail
• date
• kubectl
• jq (for plotting)

Quickstart

Choose two nodes to act as server/client on your cluster (for example node1 and node2) . Then start the knb :

./knb --verbose --client-node node1 --server-node node2

If you omit the --verbose flag, it will also complete, but you will have no output until the end of the benchmark.

Docker quickstart

Environment variables:

• NODE_AUTOSELECT Auto-selects a few nodes from cluster for running tests
• MASTER_ELIGIBLE Master nodes can also be chosen

You need to mount a valid kubeconfig inside the container and provide all other required flags to knb:

docker run -e NODE_AUTOSELECT=1 -it --hostname knb --name knb --rm -v /home/user/my-graphs:/my-graphs -v /path/to/my/kubeconfig:/.kube/config olegeech/k8s-bench-suite --verbose --plot --plot-dir /my-graphs

Examples

• Simple benchmark from "node1" to "node2" in verbose mode :

  knb -v -cn node1 -sn node2

• Benchmark from "nA" to "nB" and save data in file mybench.knbdata

  knb -cn nA -sn nB -o data -f mybench.knbdata

• Generate report in json from previous benchmark data file mybench.knbdata

  knb -fd mybench.knbdata -o json

• Plot graphs from previous benchmark data file mybench.knbdata

  knb -fd mybench.knbdata --plot --plot-args '--width 900 --height 600'

• To run benchmark from node A to node B, showing only result in yaml format :

  knb -cn A -sn B -o yaml

• To run benchmark from node Asterix to node Obelix, with the most verbose output and a result as json in a res.json file :

  knb --debug -cn Asterix -sn Obelix -o json -f res.json

• Running benchmark in namespace myns :

  knb -n myns -cn node1 -sn node2

• Run only idle and tcp benchmark :

  knb -cn clientnode -sn servernode -ot idle,tcp

Usage

To display usage, use the -h flag :

aducastel@infrabuilder:~/k8s-bench-suite$ ./knb -h

knb is a network benchmark tool for Kubernetes CNI

There are two modes :
- benchmark mode : will actually run benchmark on a cluster
- from data mode : read data generated by previous benchmark with "-o data" flag

=====[ Benchmark mode ]====================================================

 Mandatory flags :

    -cn <nodename>
    --client-node <nodename>    : Define kubernetes node name that will host the client part

    -sn <nodename>
    --server-node <nodename>    : Define kubernetes node name that will host the server part

 Optionnal flags :
    -d <time-in-scd>
    --duration <time-in-scd>    : Set the benchmark duration for each test in seconds (Default 10)

    -k
    --keep                      : Keep data directory instead of cleaning it (tmp dir that contains raw benchmark data)

    -n <namespace>
    --namespace <namespace>     : Set the target kubernetes namespace

    --name <name>               : Set the name of this benchmark run

    -ot <testlist>
    --only-tests <testlist>     : Only run a subset of benchmark tests, comma separated (Ex: -ot tcp,idle)
                                  Possible values: all, tcp, udp, p2p, p2s , p2ptcp, p2pudp, p2stcp, p2sudp, idle

    -sbs <size>
    --socket-buffer-size <size> : Set the UDP socket buffer size with unit, or 'auto'. ex: '256K' (Default: auto)

    -t <time-in-scd>
    --timeout <time-in-scd>     : Set the pod ready wait timeout in seconds (Default 30)

=====[ From Data mode ]====================================================

Mandatory flags :
    -fd <path>
    --from-data <path>          : Define the path to the data to read from
                                  Data file must be rendered with '--output data'

=====[ Common optionnal flags ]============================================

    --debug                     : Set the debug level to "debug"

    -dl <level>
    --debug-level <level>       : Set the debug level
                                  Possible values: standard, warn, info, debug

    -f <filepath>
    --file <filepath>           : Set the output file

    -h
    --help                      : Display this help message

    -p
    --plot                      : Plot data using plotly/orca

    -pd
    --plot-dir                  : Directory where to save graphs
                                  Defaults to the current directory

    -pa
    --plot-args                 : Arguments to the plotly's 'orca graph' function
                                  Defaults to '--width 900 --height 500'

    -o <format>
    --output <format>           : Set the output format. Defaults to 'text'
                                  Possible values: text, yaml, json, data
    -v
    --verbose                   : Activate the verbose mode by setting debug-level to 'info'

    -V
    --version                   : Show current script version

=====[ Examples ]==========================================================

  Simple benchmark from "node1" to "node2" in verbose mode
  -------------------------------------------------------------------------
  | knb -v -cn node1 -sn node2                                            |
  -------------------------------------------------------------------------

  Benchmark from "nA" to "nB" with data saved in file "mybench.knbdata"
  -------------------------------------------------------------------------
  | knb -cn nA -sn nB -o data -f mybench.knbdata                          |
  -------------------------------------------------------------------------

  Generate report in json from previous benchmark file "mybench.knbdata"
  -------------------------------------------------------------------------
  | knb -fd mybench.knbdata -o json                                       |
  -------------------------------------------------------------------------

  Create graph images from previous benchmark file "mybench.knbdata"
  -------------------------------------------------------------------------
  | knb -fd mybench.knbdata  --plot --plot-dir ./my-graphs                |
  -------------------------------------------------------------------------

  Run only idle and tcp benchmark :
  -------------------------------------------------------------------------
  | knb -cn clientnode -sn servernode -ot idle,tcp                        |
  -------------------------------------------------------------------------

Graph examples