[Feature] CNF Installation (5), Prepare tests for new installation method #2153

kosstennbl · 2024-09-19T13:37:13Z

Some of the tests in testsuite are not compatible with planned new installation method. They should be redesigned or given skip conditions if not suitable for some situations.

Potential issues:

Tests that are specific to install method
Tests that don't support multiple namespaces
Tests that use/replicate current install methods
Tests that use configmaps, created outside of the test

List of problematic tests (possibly incomplete) and ideas for problem solutions:

5g_validator.cr: Not found
compatibility.cr:
- "helm_chart_valid", "helm_chart_published", "helm_deploy": Check only through helm_charts and helm_dirs deployment types (where applicable).
- "cni_compatible": Check if cnf_to_new_cluster process is suitable for new installation method. Ensure correct test logic.
configuration.cr:
- Kyverno and its policies should work fine, they are applied cluster-wide
- "ip_addresses" should be removed, "hardcoded_ip_addresses_in_k8s_runtime_configuration" should look through common manifest
- "alpha_k8s_apis" cnf_to_new_cluster usage, needs checking.
microservice.cr:
- "reasonable_startup_time": uses configmap that is created on installation.
- "service_discovery": dependent on using single namespace for resource kind name discovery.
observability.cr:
- "prometheus_traffic", "open_metrics": uses release_name to create configmap during test. Configmap should be renamed.
- "tracing": uses configmap that is created on installation.
ran.cr:
- "oran_e2_connection": uses configmap that is created on installation.
reliability.cr:
- Litmus tests don't seem to create problems in the scope of this epic. Paths for writing experiment results should be changed.
security.cr:
- Kubescape scans are done cluster-wide, should be ok.

- open_metrics and prometheus_traffic - changed config map naming - ip_addresses - removed Refs: #2153 Signed-off-by: barmull barbora.muller@tietoevry.com

- open_metrics and prometheus_traffic - change name of config maps - ip_addresses - removed - hardcoded_ip_addresses_in_k8s_runtime_configuration - change to use common_manifest.yml file Refs: #2153 Signed-off-by: barmull barbora.muller@tietoevry.com

Installation configmap can significantly complicate new installation process and has multiple issues with its design (and related tests). Remove installation congigmap Redesign related tests or make them always skip until further redesign would be done in the scope of different change Refs: #2153 Signed-off-by: Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>

kosstennbl · 2024-10-07T13:19:00Z

Two tests were disabled as a part of 5.2 (#2164): 'tracing' and 'oran_e2_connection'.
The reason is their usage of installation configmap.
In my opinion, installation configmaps bring multiple problems to the project:

Interdependence of installation process and testing process. Making testing and installation independent from each other allows for easier development, clearer code, and opens possibility to run tests on non-testsuite installed CNFs in future.
Bloating of installation code. Install process is much simpler without additional tracking for Jaeger and e2 connections. I think that, unless absolutely necessary, adding non-install functionality to the install process should be avoided.
Unnecessarily creating K8s resources. To me - it seems that there is no reason for these installation metrics to be a configmap. That could be solved by using temporary files in future, in the scope of redesigning these tests (if adding installation-specific metrics would be necessary)

Separate ticket should be created for redesign and return of these tests.

Installation configmap can significantly complicate new installation process and has multiple issues with its design (and related tests). Remove installation congigmap Redesign related tests or make them always skip until further redesign would be done in the scope of different change Refs: #2153 Signed-off-by: Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>

kosstennbl · 2024-10-08T09:50:20Z

Additional issue was discovered: we need to somehow ensure correct namespaces for each resource for cnf_workload_resource method. If deployments in CNF would be on different namespaces, get_deployment_namespace method wouldn't work.

Add: Being done in the scope of (4)

- open_metrics and prometheus_traffic - change name of config maps - ip_addresses - removed - hardcoded_ip_addresses_in_k8s_runtime_configuration - change to use common_manifest.yml file Refs: #2153 Signed-off-by: barmull barbora.muller@tietoevry.com

Installation configmap can significantly complicate new installation process and has multiple issues with its design (and related tests). Remove installation congigmap Redesign related tests or make them always skip until further redesign would be done in the scope of different change Refs: #2153 Signed-off-by: Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>

Temporarily disable cnf_to_new_cluster tests as this process needs to be redesigned to new installation format. Also, adapt service_discovery to namespaced manifest generation. Refs: #2153 Signed-off-by: Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>

Refs: #2153 - Removed functions that will not be available with new installation process. - Tests will have to be updated further once the new points system is introduced. Signed-off-by: svteb <slavo.valko@tietoevry.com>

Refs: #2153 - Removed functions that will not be available with new installation process. - Tests now account for multiple deployments in one CNF and check accordingly. In case of any failure the whole task exists with FAILED, additional information about which chart failed can be found in the logs. Signed-off-by: svteb <slavo.valko@tietoevry.com>