Feat/sonobuoy integration

[mbuechse]

Rebased feat/sonobuoy-integration on main

[tonifinger]

I tested it over night with two test subjects kind-current,
and kind-current-1:
Keep in mind I commend out all test apart from cncf-k8s-conformance in scs-compatible-kaas.yaml for testing only this long running test.

(venv-sono) tf@tf-ThinkPad:~/repos/ci/scs/standards/Tests$ time ./scs-test-runner.py --config ./config.toml --debug run --preset="kaas-dev" --monitor-url localhost -o REPORT_kind.yaml
DEBUG: running tests for scope(s) scs-compatible-kaas and subject(s) kind-current, kind-current-1
DEBUG: monitor url: localhost, num_workers: 1, output: REPORT_kind.yaml
DEBUG: running "./kaas/sonobuoy_handler/run_sonobuoy.py -k kind-current-1/kubeconfig.yaml -r kind-current-1/sono-results -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance'"...
DEBUG: kubeconfig: kind-current-1/kubeconfig.yaml 
DEBUG: working from /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests
DEBUG: placing results at kind-current-1/sono-results
DEBUG: sonobuoy executable at /home/tf/.local/bin/sonobuoy
INFO: running sonobuoy for testcase cncf-k8s-conformance
DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current-1/kubeconfig.yaml run --wait --mode=certified-conformance
DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current-1/kubeconfig.yaml status --json
DEBUG: plugin:e2e:failed
DEBUG: plugin:systemd-logs:passed
DEBUG: plugin:systemd-logs:passed
ERROR: sonobuoy reports 374 passed, 34 failed, 6797 skipped
DEBUG: .. rc 3, 0 critical, 1 error
********************************************************************************
kind-current-1 SCS-compatible KaaS v1 (draft):
- main: FAIL (0 passed, 1 failed)
  - FAILED:
    - cncf-k8s-conformance:
DEBUG: running "./kaas/sonobuoy_handler/run_sonobuoy.py -k kind-current/kubeconfig.yaml -r kind-current/sono-results -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance'"...
DEBUG: kubeconfig: kind-current/kubeconfig.yaml 
DEBUG: working from /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests
DEBUG: placing results at kind-current/sono-results
DEBUG: sonobuoy executable at /home/tf/.local/bin/sonobuoy
INFO: running sonobuoy for testcase cncf-k8s-conformance
DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current/kubeconfig.yaml run --wait --mode=certified-conformance
DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current/kubeconfig.yaml status --json
DEBUG: plugin:systemd-logs:passed
DEBUG: plugin:systemd-logs:passed
DEBUG: plugin:e2e:failed
ERROR: sonobuoy reports 376 passed, 34 failed, 6199 skipped
DEBUG: .. rc 3, 0 critical, 1 error
********************************************************************************
kind-current SCS-compatible KaaS v1 (draft):
- main: FAIL (0 passed, 1 failed)
  - FAILED:
    - cncf-k8s-conformance:

real	341m2,338s
user	0m8,174s
sys	0m2,493s

The content of the report file:

---
spec:
  uuid: 1fffebe6-fd4b-44d3-a36c-fc58b4bb0180
  name: SCS-compatible KaaS
  url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-kaas.yaml
checked_at: 2024-11-21 05:27:39.169580
reference_date: 2024-11-21
subject: kind-current
versions:
  v1:
    cncf-k8s-conformance:
      result: -1
      invocation: fddb694d-cfe4-4c6c-ac12-de88b43eceee
run:
  uuid: 2b8e2486-ac2a-44a7-8459-e2a5becc75ba
  argv:
  - /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests/./scs-compatible-kaas.yaml
  - --debug
  - -C
  - -o
  - /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests/tmp9_wmdbw0/report-0.yaml
  - -s
  - kind-current
  - -a
  - os_cloud=kind-current
  - -a
  - subject_root=kind-current
  assignment:
    os_cloud: kind-current
    subject_root: kind-current
  sections: null
  forced_version: null
  forced_tests: null
  invocations:
    fddb694d-cfe4-4c6c-ac12-de88b43eceee:
      id: fddb694d-cfe4-4c6c-ac12-de88b43eceee
      cmd: ./kaas/sonobuoy_handler/run_sonobuoy.py -k kind-current/kubeconfig.yaml
        -r kind-current/sono-results -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance'
      result: 0
      results:
        cncf-k8s-conformance: -1
      rc: 3
      stdout:
      - 'cncf-k8s-conformance: FAIL'
      stderr:
      - 'DEBUG: kubeconfig: kind-current/kubeconfig.yaml '
      - 'DEBUG: working from /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests'
      - 'DEBUG: placing results at kind-current/sono-results'
      - 'DEBUG: sonobuoy executable at /home/tf/.local/bin/sonobuoy'
      - 'INFO: running sonobuoy for testcase cncf-k8s-conformance'
      - 'DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current/kubeconfig.yaml
        run --wait --mode=certified-conformance'
      - 'DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current/kubeconfig.yaml
        status --json'
      - 'DEBUG: plugin:systemd-logs:passed'
      - 'DEBUG: plugin:systemd-logs:passed'
      - 'DEBUG: plugin:e2e:failed'
      - 'ERROR: sonobuoy reports 376 passed, 34 failed, 6199 skipped'
      info: 1
      warning: 0
      error: 1
      critical: 0
---
spec:
  uuid: 1fffebe6-fd4b-44d3-a36c-fc58b4bb0180
  name: SCS-compatible KaaS
  url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-kaas.yaml
checked_at: 2024-11-21 02:33:58.212539
reference_date: 2024-11-20
subject: kind-current-1
versions:
  v1:
    cncf-k8s-conformance:
      result: -1
      invocation: 0f7f3507-4e01-4170-8e08-a310513dbd9a
run:
  uuid: f7bff533-29a6-4659-8e6e-6cb051094d69
  argv:
  - /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests/./scs-compatible-kaas.yaml
  - --debug
  - -C
  - -o
  - /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests/tmp9_wmdbw0/report-1.yaml
  - -s
  - kind-current-1
  - -a
  - os_cloud=kind-current-1
  - -a
  - subject_root=kind-current-1
  assignment:
    os_cloud: kind-current-1
    subject_root: kind-current-1
  sections: null
  forced_version: null
  forced_tests: null
  invocations:
    0f7f3507-4e01-4170-8e08-a310513dbd9a:
      id: 0f7f3507-4e01-4170-8e08-a310513dbd9a
      cmd: ./kaas/sonobuoy_handler/run_sonobuoy.py -k kind-current-1/kubeconfig.yaml
        -r kind-current-1/sono-results -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance'
      result: 0
      results:
        cncf-k8s-conformance: -1
      rc: 3
      stdout:
      - 'cncf-k8s-conformance: FAIL'
      stderr:
      - 'DEBUG: kubeconfig: kind-current-1/kubeconfig.yaml '
      - 'DEBUG: working from /home/tf/repos/ci/scs/01_ISSUES/standard_SONOBUOY_PR_832/Tests'
      - 'DEBUG: placing results at kind-current-1/sono-results'
      - 'DEBUG: sonobuoy executable at /home/tf/.local/bin/sonobuoy'
      - 'INFO: running sonobuoy for testcase cncf-k8s-conformance'
      - 'DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current-1/kubeconfig.yaml
        run --wait --mode=certified-conformance'
      - 'DEBUG: invoking /home/tf/.local/bin/sonobuoy --kubeconfig kind-current-1/kubeconfig.yaml
        status --json'
      - 'DEBUG: plugin:e2e:failed'
      - 'DEBUG: plugin:systemd-logs:passed'
      - 'DEBUG: plugin:systemd-logs:passed'
      - 'ERROR: sonobuoy reports 374 passed, 34 failed, 6797 skipped'
      info: 1
      warning: 0
      error: 1
      critical: 0

So in conclusion, the sonobuoy integration together with the cncf conformance tests does work.

[tonifinger]

LGTM!