Feat/sonobuoy integration

[tonifinger]

This PR introduces the usage of sonobuoy to

• Run kubernetes e2e tests
• Parse results from the sonobouy testsuite

[tonifinger]

I performed the cncf conformance tests using the moin cluster with the following workload cluster:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: scs-cluster-129
  labels:
    managed-secret: cloud-config
spec:
  topology:
    variables:
    class: openstack-scs-1-29-v4
    controlPlane:
      replicas: 3
    version: v1.29.10
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 3

I manually downloaded the kubeconfig.yaml for the workload cluster and used the static_plugin to run the scs-conformance tests against the cluster:

(venv) tf@tf-ThinkPad:~/standards/Tests$ time ./scs-test-runner.py --config ./config.toml --debug run --preset="all-kaas" --monitor-url localhost -o REPORT.yaml
DEBUG: running tests for scope(s) scs-compatible-kaas and subject(s) cspB-static
DEBUG: monitor url: localhost, num_workers: 4, output: REPORT.yaml
DEBUG: running "./kaas/sonobuoy_handler/run_sonobuoy.py -k cspB-static/kubeconfig.yaml -r cspB-static/sono-results -c 'cncf-conformance'  -a '--mode=certified-conformance'"...
ERROR: failed: 4
ERROR: skipped: 7022
ERROR: 388 passed, 4 failed, 7022 skipped
DEBUG: .. rc 3, 0 critical, 3 error
WARNING: ignoring invalid result id: cncf-conformance
********************************************************************************
cspB-static SCS-compatible KaaS v1 (draft):
- main: TENTATIVE pass (0 passed, 1 missing)
  - MISSING:
    - cncf-k8s-conformance:
4.94user 1.45system 2:04:23elapsed 0%CPU (0avgtext+0avgdata 59796maxresident)k
88592inputs+64outputs (265major+29770minor)pagefaults 0swaps

As you can see by the output the cluster did not pass the cncf conformance tests.

REPORT.yaml:

cat REPORT.yaml
---
spec:
  uuid: 1fffebe6-fd4b-44d3-a36c-fc58b4bb0180
  name: SCS-compatible KaaS
  url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-kaas.yaml
checked_at: 2024-11-19 01:33:03.410632
reference_date: 2024-11-18
subject: cspB-static
versions:
  v1: {}
run:
  uuid: a2484705-da2b-4ca3-abfe-d92b8ed733b8
  argv:
  - /home/tf/repos/ci/scs/01_ISSUES/standard_710/Tests/./scs-compatible-kaas.yaml
  - --debug
  - -C
  - -o
  - /home/tf/repos/ci/scs/01_ISSUES/standard_710/Tests/tmpqs_qlpiq/report-0.yaml
  - -s
  - cspB-static
  - -a
  - os_cloud=cspB-static
  - -a
  - subject_root=cspB-static
  assignment:
    os_cloud: cspB-static
    subject_root: cspB-static
  sections: null
  forced_version: null
  forced_tests: null
  invocations:
    6c5669e7-048a-495c-ba26-e4dfa199d407:
      id: 6c5669e7-048a-495c-ba26-e4dfa199d407
      cmd: ./kaas/sonobuoy_handler/run_sonobuoy.py -k cspB-static/kubeconfig.yaml
        -r cspB-static/sono-results -c 'cncf-conformance'  -a '--mode=certified-conformance'
      result: 0
      results:
        cncf-conformance: -1
      rc: 3
      stdout:
      - 'Sonobuoy Version: v0.57.2'
      - 'MinimumKubeVersion: 1.17.0'
      - 'MaximumKubeVersion: 1.99.99'
      - 'GitSHA: cc22d58f4c8b5a155bd1778cd3702eca5ad43e05'
      - 'GoVersion: go1.22.4'
      - 'Platform: linux/amd64'
      - 'API Version:  v1.29.10'
      - check kubeconfig
      - plugin:e2e:failed
      - plugin:systemd-logs:passed
      - plugin:systemd-logs:passed
      - plugin:systemd-logs:passed
      - plugin:systemd-logs:passed
      - plugin:systemd-logs:passed
      - plugin:systemd-logs:passed
      - 'cncf-conformance: FAIL'
      stderr:
      - 'ERROR: failed: 4'
      - 'ERROR: skipped: 7022'
      - 'ERROR: 388 passed, 4 failed, 7022 skipped'
      info: 0
      warning: 0
      error: 3
      critical: 0

[mbuechse]

To be continued at #832