[BUG]: cert-csi CapacityTracking test fails when more than 1 CSI driver is deployed

[dancohen21]

Bug Description

Environment:
• cert-csi release 1.5.0
• PowerStore 3.6.1.2 connected with ISCSI
• OpenShift 4.14 using csm-operator 1.5.1 and csi-powerstore 2.10.1
• using the powerstore sample on https://dell.github.io/csm-docs/docs/support/cert-csi/

cert-csi CapacityTracking test fails when more than 1 CSI driver is deployed.
The environment I was using had 3 CSI drivers deployed. Unity, Isilon, and PowerStore.

the code path for CapacityTracking in cert-csi did not work as expected and failed to pass the test.
possibly: topologiesCount = 3,
https://github.com/dell/cert-csi/blob/859ad6d26e0c88214430391f6f901af6d29ee4fe/pkg/testcore/suites/functional-suites.go#L869C67-L869C82
does not match createdCount = 1
https://github.com/dell/cert-csi/blob/859ad6d26e0c88214430391f6f901af6d29ee4fe/pkg/k8sclient/resources/csistoragecapacity/csistoragecapacity.go#L129

Removing the Unity and Isilon CSM instance / CSI Drivers, and only having the PowerStore CSI Driver deployed,
the CapacityTracking test passed as expected.

Organizations who use multiple Dell storage systems will have multiple Dell CSI drivers deployed. The cert-csi test suite needs to accommodate this configuration accordingly.

Logs

with 3 Dell CSI drivers deployed

[root@ocpcsi-bastion-10-241-171-47 cleanrun-sq]# ../cert-csi-linux-amd64 functional-test capacity-tracking --sc powerstore-nfs-waitforfirstconsumer --drns powerstore
[2024-10-04 11:40:52] INFO Starting cert-csi; ver. 1.5.0
[2024-10-04 11:40:52] INFO Using EVENT observer type
[2024-10-04 11:40:52] INFO Using default config
[2024-10-04 11:40:52] INFO Successfully loaded config. Host: https://console-openshift-console.apps.ocpcsi.powerflex.lab.dell.com:6443
[2024-10-04 11:40:52] INFO Created new KubeClient
[2024-10-04 11:40:52] INFO Found 3 topology segment(s) in csinode
[2024-10-04 11:40:52] INFO Using default image: docker.io/centos:latest
[2024-10-04 11:40:52] INFO Creating capacity-tracking-27291686 storage class
[2024-10-04 11:40:52] INFO Waiting for CSIStorageCapacity to be CREATED for capacity-tracking-27291686 storage class in powerstore namespace
[2024-10-04 11:45:52] ERROR Suite CapacityTrackingSuite failed; error=timed out waiting for the condition
[2024-10-04 11:45:57] INFO FAILURE: CapacityTrackingSuite in 5m5.524631743s
[2024-10-04 11:45:57] INFO Trying to connect to cluster...
[2024-10-04 11:45:57] INFO Created new KubeClient
[2024-10-04 11:45:57] FATAL During this run 0.0% of suites succeeded

with 1 Dell CSI driver deployed

Passing CapacityTracking test for NFS storage class
[2024-10-04 13:07:53] INFO Starting cert-csi; ver. 1.5.0
[2024-10-04 13:07:53] INFO Using EVENT observer type
[2024-10-04 13:07:53] INFO Using default config
[2024-10-04 13:07:53] INFO Successfully loaded config. Host: https://console-openshift-console.apps.ocpcsi.powerflex.lab.dell.com:6443/
[2024-10-04 13:07:53] INFO Created new KubeClient
[2024-10-04 13:07:53] INFO Found 1 topology segment(s) in csinode
[2024-10-04 13:07:53] INFO Using default image: docker.io/centos:latest
[2024-10-04 13:07:53] INFO Creating capacity-tracking-ea4578cb storage class
[2024-10-04 13:07:53] INFO Waiting for CSIStorageCapacity to be CREATED for capacity-tracking-ea4578cb storage class in powerstore namespace
[2024-10-04 13:07:55] INFO All CSIStorageCapacity are created in 2.016635363s
[2024-10-04 13:07:55] INFO Deleting capacity-tracking-ea4578cb storage class,
[2024-10-04 13:07:55] INFO Waiting for CSIStorageCapacity to be DELETED for capacity-tracking-ea4578cb storage class in powerstore namespace
[2024-10-04 13:07:58] INFO All CSIStorageCapacity are deleted in 2.016172318s
[2024-10-04 13:07:58] INFO Updating CSIStorageCapacity for powerstore-nfs-waitforfirstconsumer storage class, setting capacity to 0
[2024-10-04 13:07:58] INFO Creating capacity-tracking-pod-edfccdf3 pod using powerstore-nfs-waitforfirstconsumer storage class
[2024-10-04 13:08:03] INFO capacity-tracking-pod-edfccdf3 pod is PENDING
[2024-10-04 13:08:03] INFO Waiting for provisioner to POLL GetCapacity for powerstore-nfs-waitforfirstconsumer storage class in 5m0s
[2024-10-04 13:12:51] INFO Provisioner POLLED GetCapacity for powerstore-nfs-waitforfirstconsumer storage class
[2024-10-04 13:12:51] INFO Waiting for pod capacity-tracking-pod-edfccdf3 to be READY
[2024-10-04 13:12:51] INFO capacity-tracking-pod-edfccdf3 pod is RUNNING
[2024-10-04 13:12:54] INFO SUCCESS: CapacityTrackingSuite in 5m0.663340391s
[2024-10-04 13:12:54] INFO Trying to connect to cluster...
[2024-10-04 13:12:54] INFO Created new KubeClient
[2024-10-04 13:12:54] INFO During this run 100.0% of suites succeeded

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

run cert-csi CapacityTracking test
./cert-csi-linux-amd64 functional-test capacity-tracking --sc powerstore-nfs-waitforfirstconsumer --drns powerstore
when more than 1 CSIDriver is deployed (Isilon, Unity, PowerMax, PowerStore, etc)

Expected Behavior

hypothesis:

the CapacityTracking test relies on getTopologyCount
topologiesCount, err := getTopologyCount()

This function does not consider the specific storage class and CSI driver being tested, and results in too many Items to campare against later in the code.

func getTopologyCount() (int, error) {
exe := []string{"bash", "-c", "kubectl describe csinode | grep 'Topology Keys'"}
str, err := FindDriverLogs(exe)
if err != nil {
return 0, err
}
topologies := strings.Split(strings.TrimSpace(strings.ReplaceAll(str, "Topology Keys:", "")), "\n")
topologies = removeDuplicates(topologies)
topologiesCount := len(topologies)
return topologiesCount, nil
}

CSM Driver(s)

CSM 1.10
csi-powerstore 2.10.1

Installation Type

CSM-Operator 1.5.1

Container Storage Modules Enabled

No response

Container Orchestrator

OpenShift 4.14

Operating System

OpenShift 4.14

[csmbot]

@dancohen21: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and respond appropriately.

---

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at container.storage.modules@dell.com.

[jooseppi-luna]

/sync

[csmbot]

link: 29804