Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Powermax to handle Minimal Manifests with Mount Credentials #872

Merged
merged 6 commits into from
Jan 27, 2025
Merged

Update Powermax to handle Minimal Manifests with Mount Credentials #872

merged 6 commits into from
Jan 27, 2025

Conversation

falfaroc
Copy link
Contributor

@falfaroc falfaroc commented Jan 22, 2025 •
edited by ashleyvjoy
Loading

Description

Fix dynamically adding parameters for the config map and mount credentials approach for Powermax. Also, fix the processing of the minimal manifest installation for Powermax to work with the mount credentials.

GitHub Issues

List the GitHub issues impacted by this PR:

GitHub Issue #
dell/csm#1614

Checklist:

  • I have performed a self-review of my own code to ensure there are no formatting, vetting, linting, or security issues
  • I have verified that new and existing unit tests pass locally with my changes
  • I have not allowed coverage numbers to degenerate
  • I have maintained at least 90% code coverage
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have maintained backward compatibility
  • I have executed the relevant end-to-end test scenarios

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

  • Ensure that unit tests are still passing as expected
-> # make test
test -s /root/git/public/dell/csm-operator/bin/controller-gen && /root/git/public/dell/csm-operator/bin/controller-gen --version | grep -q v0.16.5 || \
GOBIN=/root/git/public/dell/csm-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.16.5
/root/git/public/dell/csm-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/git/public/dell/csm-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
(cd core; rm -f core_generated.go; go generate)
go run core/semver/semver.go -f mk > semver.mk
go fmt ./...
core/core_generated.go
go vet ./...
KUBEBUILDER_ASSETS="/root/.local/share/kubebuilder-envtest/k8s/1.31.0-linux-amd64" go test ./... -coverprofile cover.out
        github.com/dell/csm-operator            coverage: 0.0% of statements
        github.com/dell/csm-operator/api/v1             coverage: 0.0% of statements
        github.com/dell/csm-operator/core/semver                coverage: 0.0% of statements
        github.com/dell/csm-operator/core               coverage: 0.0% of statements
?       github.com/dell/csm-operator/pkg/constants      [no test files]
        github.com/dell/csm-operator/pkg/logger         coverage: 0.0% of statements
        github.com/dell/csm-operator/tests/shared/clientgoclient                coverage: 0.0% of statements
        github.com/dell/csm-operator/tests/shared               coverage: 0.0% of statements
        github.com/dell/csm-operator/tests/shared/crclient              coverage: 0.0% of statements
ok      github.com/dell/csm-operator/controllers        57.151s coverage: 86.4% of statements
ok      github.com/dell/csm-operator/k8s        0.228s  coverage: 90.3% of statements
ok      github.com/dell/csm-operator/pkg/drivers        1.558s  coverage: 96.9% of statements
ok      github.com/dell/csm-operator/pkg/modules        4.351s  coverage: 90.2% of statements
ok      github.com/dell/csm-operator/pkg/resources/configmap    0.074s  coverage: 100.0% of statements
ok      github.com/dell/csm-operator/pkg/resources/csidriver    0.105s  coverage: 100.0% of statements
ok      github.com/dell/csm-operator/pkg/resources/daemonset    0.190s  coverage: 100.0% of statements
ok      github.com/dell/csm-operator/pkg/resources/deployment   0.117s  coverage: 100.0% of statements
ok      github.com/dell/csm-operator/pkg/resources/rbac 0.100s  coverage: 100.0% of statements
ok      github.com/dell/csm-operator/pkg/resources/serviceaccount       0.148s  coverage: 100.0% of statements
ok      github.com/dell/csm-operator/pkg/utils  3.172s  coverage: 82.0% of statements
  • Ensure Powermax E2E tests run successfully
-> # bash run-e2e-test.sh --pmax
/root/git/public/dell/csm-operator/tests/e2e
  W0122 20:38:26.872454    2727 test_context.go:541] Unable to find in-cluster config, using default host : https://127.0.0.1:6443
  I0122 20:38:26.880340 2727 test_context.go:564] The --provider flag is not set. Continuing as if --provider=skeleton had been used.
Running Suite: CSM Operator End-to-End Tests - /root/git/public/dell/csm-operator/tests/e2e
===========================================================================================
Random Seed: 1737578296

Will run 1 of 1 specs
------------------------------
[BeforeSuite]
/root/git/public/dell/csm-operator/tests/e2e/e2e_test.go:98
  STEP: Getting test environment variables @ 01/22/25 20:38:26.885
  STEP: [powermax] @ 01/22/25 20:38:26.885
  STEP: Reading values file @ 01/22/25 20:38:26.885
  STEP: Getting a k8s client @ 01/22/25 20:38:27.453
[BeforeSuite] PASSED [0.573 seconds]
------------------------------
[run-e2e-test] E2E Testing Running all test Given Test Scenarios
/root/git/public/dell/csm-operator/tests/e2e/e2e_test.go:139
  STEP: Starting: Install PowerMax Driver(Standalone)  @ 01/22/25 20:38:27.457
  STEP: Returning false here @ 01/22/25 20:38:27.457
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 20:38:27.457
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:38:27.548
  STEP:      Executing  Set up reverse proxy tls secret namespace [powermax] @ 01/22/25 20:38:28.755
revproxy-certs secret already exists, skipping creation.
csirevproxy-tls-secret already exists, skipping creation.
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:38:29.234
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 20:38:29.947
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:38:30.88
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 20:38:31.733
  I0122 20:38:31.734538 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 20:38:32.425762 2727 builder.go:146] stderr: ""
  I0122 20:38:32.425821 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/22/25 20:38:32.425
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 20:38:52.51
  STEP:      Executing  Run custom test @ 01/22/25 20:39:12.586
  I0122 20:39:12.586888 2727 util.go:650] Running cert-csi [test vio --sc op-e2e-pmax --chainLength 1 --chainNumber 1]
[2025-01-22 20:39:12]  INFO Starting cert-csi; ver. 1.4.1
[2025-01-22 20:39:12]  INFO Using EVENT observer type
[2025-01-22 20:39:12]  INFO Using config from /root/.kube/config
[2025-01-22 20:39:12]  INFO Successfully loaded config. Host: https://10.227.248.175:6443
[2025-01-22 20:39:12]  INFO Created new KubeClient
[2025-01-22 20:39:12]  INFO Running 1 iteration(s)
[2025-01-22 20:39:12]  INFO     *** ITERATION NUMBER 1 ***
[2025-01-22 20:39:12]  INFO Starting VolumeIoSuite with op-e2e-pmax storage class
[2025-01-22 20:39:12]  INFO Successfully created namespace volumeio-test-91a7f60d
[2025-01-22 20:39:12]  INFO Using default number of volumes
[2025-01-22 20:39:12]  INFO Using default image: docker.io/centos:latest
[2025-01-22 20:39:12]  INFO Creating IO pod
[2025-01-22 20:39:12]  INFO Waiting for pod iowriter-test-89m9w to be READY
[2025-01-22 20:39:57]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2025-01-22 20:39:58]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2025-01-22 20:40:00]  INFO Waiting until no Volume Attachments with PV  left
[2025-01-22 20:40:00]  INFO VolumeAttachment deleted
[2025-01-22 20:40:00]  INFO Deleting all resources in namespace volumeio-test-91a7f60d
[2025-01-22 20:40:12]  INFO Namespace volumeio-test-91a7f60d was deleted in 12.089312999s
[2025-01-22 20:40:13]  INFO SUCCESS: VolumeIoSuite in 1m0.204405026s
[2025-01-22 20:40:13]  INFO Started generating reports...
Collecting metrics
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-22 20:40:13]  INFO Started generating reports...
Collecting metrics
Generating plots
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-22 20:40:13]  WARN No ResourceUsageMetrics provided
[2025-01-22 20:40:13] ERROR no ResourceUsageMetrics provided
report-test-run-b9aa15a5:
Name: test-run-b9aa15a5
Host: https://10.227.248.175:6443
StorageClass: op-e2e-pmax
Minimum and Maximum EntityOverTime charts:

/root/.cert-csi/reports/test-run-b9aa15a5/PodsCreatingOverTime.png

/root/.cert-csi/reports/test-run-b9aa15a5/PodsReadyOverTime.png

/root/.cert-csi/reports/test-run-b9aa15a5/PodsTerminatingOverTime.png

/root/.cert-csi/reports/test-run-b9aa15a5/PvcsCreatingOverTime.png

/root/.cert-csi/reports/test-run-b9aa15a5/PvcsBoundOverTime.png

Tests:
--------------------------------------------------------------
1. TestCase: VolumeIoSuite
            Started:   2025-01-22 20:39:12.780916685 +0000 UTC
            Ended:     2025-01-22 20:40:13.029299577 +0000 UTC
            Result:    SUCCESS

            Stage metrics:
                    PVCAttachment:
                        Avg: 4.472952705s
                        Min: 4.472952705s
                        Max: 4.472952705s
                        Histogram:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCAttachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCAttachment_boxplot.png
                    PVCBind:
                        Avg: 2.93622335s
                        Min: 2.93622335s
                        Max: 2.93622335s
                        Histogram:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCBind.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCBind_boxplot.png
                    PVCCreation:
                        Avg: 7.610314378s
                        Min: 7.610314378s
                        Max: 7.610314378s
                        Histogram:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCCreation_boxplot.png
                    PVCDeletion:
                        Avg: 99.913218ms
                        Min: 99.913218ms
                        Max: 99.913218ms
                        Histogram:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCDeletion_boxplot.png
                    PVCUnattachment:
                        Avg: 1.744060156s
                        Min: 1.744060156s
                        Max: 1.744060156s
                        Histogram:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCUnattachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PVCUnattachment_boxplot.png
                    PodCreation:
                        Avg: 43.223200866s
                        Min: 43.223200866s
                        Max: 43.223200866s
                        Histogram:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PodCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PodCreation_boxplot.png
                    PodDeletion:
                        Avg: 863.474757ms
                        Min: 863.474757ms
                        Max: 863.474757ms
                        Histogram:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PodDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/PodDeletion_boxplot.png
                        EntityNumberOverTime:
        /root/.cert-csi/reports/test-run-b9aa15a5/VolumeIoSuite9/EntityNumberOverTime.png

[2025-01-22 20:40:13]  INFO Avg time of a run:   47.93s
[2025-01-22 20:40:13]  INFO Avg time of a del:   12.09s
[2025-01-22 20:40:13]  INFO Avg time of all:     60.20s
[2025-01-22 20:40:13]  INFO During this run 100.0% of suites succeeded
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/22/25 20:40:13.508
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 20:40:13.599
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/22/25 20:40:13.65

err: found the following pods: csipowermax-reverseproxy-bbbc6c6b7-szk8r,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxAuthSidecar] @ 01/22/25 20:41:03.714
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:41:03.759
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:41:03.828
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:41:03.868
  STEP: Ending: Install PowerMax Driver(Standalone)
   @ 01/22/25 20:41:03.956
  STEP: Starting: Install PowerMax Driver with TLS (Standalone)  @ 01/22/25 20:41:08.96
  STEP: Returning false here @ 01/22/25 20:41:08.96
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 20:41:08.96
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:41:08.983
  STEP:      Executing  Set up reverse proxy tls secret with SAN namespace [powermax] @ 01/22/25 20:41:09.653
Temporary directory created at: /tmp/tls-setup1828511958
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:41:10.742
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 20:41:11.101
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:41:11.573
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 20:41:12.021
  I0122 20:41:12.022372 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 20:41:12.385201 2727 builder.go:146] stderr: ""
  I0122 20:41:12.385293 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/22/25 20:41:12.385
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 20:41:32.447
  STEP:      Executing  Run custom test @ 01/22/25 20:41:52.523
  I0122 20:41:52.523844 2727 util.go:650] Running cert-csi [test vio --sc op-e2e-pmax --chainLength 1 --chainNumber 1]
[2025-01-22 20:41:52]  INFO Starting cert-csi; ver. 1.4.1
[2025-01-22 20:41:52]  INFO Using EVENT observer type
[2025-01-22 20:41:52]  INFO Using config from /root/.kube/config
[2025-01-22 20:41:52]  INFO Successfully loaded config. Host: https://10.227.248.175:6443
[2025-01-22 20:41:52]  INFO Created new KubeClient
[2025-01-22 20:41:52]  INFO Running 1 iteration(s)
[2025-01-22 20:41:52]  INFO     *** ITERATION NUMBER 1 ***
[2025-01-22 20:41:52]  INFO Starting VolumeIoSuite with op-e2e-pmax storage class
[2025-01-22 20:41:52]  INFO Successfully created namespace volumeio-test-9bb2e2b5
[2025-01-22 20:41:52]  INFO Using default number of volumes
[2025-01-22 20:41:52]  INFO Using default image: docker.io/centos:latest
[2025-01-22 20:41:52]  INFO Creating IO pod
[2025-01-22 20:41:52]  INFO Waiting for pod iowriter-test-pkpht to be READY
[2025-01-22 20:42:32]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2025-01-22 20:42:33]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2025-01-22 20:42:36]  INFO Waiting until no Volume Attachments with PV  left
[2025-01-22 20:42:36]  INFO VolumeAttachment deleted
[2025-01-22 20:42:36]  INFO Deleting all resources in namespace volumeio-test-9bb2e2b5
[2025-01-22 20:42:48]  INFO Namespace volumeio-test-9bb2e2b5 was deleted in 12.084658229s
[2025-01-22 20:42:52]  INFO SUCCESS: VolumeIoSuite in 1m0.158279806s
[2025-01-22 20:42:52]  INFO Started generating reports...
Collecting metrics
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-22 20:42:52]  INFO Started generating reports...
Collecting metrics
Generating plots
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-22 20:42:53]  WARN No ResourceUsageMetrics provided
[2025-01-22 20:42:53] ERROR no ResourceUsageMetrics provided
report-test-run-ec3d1714:
Name: test-run-ec3d1714
Host: https://10.227.248.175:6443
StorageClass: op-e2e-pmax
Minimum and Maximum EntityOverTime charts:

/root/.cert-csi/reports/test-run-ec3d1714/PodsCreatingOverTime.png

/root/.cert-csi/reports/test-run-ec3d1714/PodsReadyOverTime.png

/root/.cert-csi/reports/test-run-ec3d1714/PodsTerminatingOverTime.png

/root/.cert-csi/reports/test-run-ec3d1714/PvcsCreatingOverTime.png

/root/.cert-csi/reports/test-run-ec3d1714/PvcsBoundOverTime.png

Tests:
--------------------------------------------------------------
1. TestCase: VolumeIoSuite
            Started:   2025-01-22 20:41:52.703725569 +0000 UTC
            Ended:     2025-01-22 20:42:52.861715435 +0000 UTC
            Result:    SUCCESS

            Stage metrics:
                    PVCAttachment:
                        Avg: 2.681673159s
                        Min: 2.681673159s
                        Max: 2.681673159s
                        Histogram:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCAttachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCAttachment_boxplot.png
                    PVCBind:
                        Avg: 3.217595162s
                        Min: 3.217595162s
                        Max: 3.217595162s
                        Histogram:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCBind.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCBind_boxplot.png
                    PVCCreation:
                        Avg: 6.783956211s
                        Min: 6.783956211s
                        Max: 6.783956211s
                        Histogram:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCCreation_boxplot.png
                    PVCDeletion:
                        Avg: 69.164719ms
                        Min: 69.164719ms
                        Max: 69.164719ms
                        Histogram:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCDeletion_boxplot.png
                    PVCUnattachment:
                        Avg: 1.890881925s
                        Min: 1.890881925s
                        Max: 1.890881925s
                        Histogram:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCUnattachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PVCUnattachment_boxplot.png
                    PodCreation:
                        Avg: 38.752454643s
                        Min: 38.752454643s
                        Max: 38.752454643s
                        Histogram:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PodCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PodCreation_boxplot.png
                    PodDeletion:
                        Avg: 1.05350824s
                        Min: 1.05350824s
                        Max: 1.05350824s
                        Histogram:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PodDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/PodDeletion_boxplot.png
                        EntityNumberOverTime:
        /root/.cert-csi/reports/test-run-ec3d1714/VolumeIoSuite10/EntityNumberOverTime.png

[2025-01-22 20:42:53]  INFO Avg time of a run:   43.94s
[2025-01-22 20:42:53]  INFO Avg time of a del:   12.09s
[2025-01-22 20:42:53]  INFO Avg time of all:     60.15s
[2025-01-22 20:42:53]  INFO During this run 100.0% of suites succeeded
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/22/25 20:42:53.351
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 20:42:53.444
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/22/25 20:42:53.498

err: found the following pods: csipowermax-reverseproxy-bbbc6c6b7-s5zh8,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxAuthSidecar] @ 01/22/25 20:43:43.579
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:43:43.634
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:43:43.683
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:43:43.71
  STEP: Ending: Install PowerMax Driver with TLS (Standalone)
   @ 01/22/25 20:43:43.8
  STEP: Starting: Install PowerMax Driver(Sidecar)  @ 01/22/25 20:43:48.804
  STEP: Returning false here @ 01/22/25 20:43:48.804
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 20:43:48.804
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:43:48.827
  STEP:      Executing  Set up reverse proxy tls secret namespace [powermax] @ 01/22/25 20:43:49.492
revproxy-certs secret already exists, skipping creation.
csirevproxy-tls-secret already exists, skipping creation.
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:43:49.763
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 20:43:50.153
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:43:50.637
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 20:43:51.126
  I0122 20:43:51.127449 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 20:43:51.484529 2727 builder.go:146] stderr: ""
  I0122 20:43:51.484584 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/22/25 20:43:51.484

err: expected custom resource status to be Succeeded. Got: Failed
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 20:44:41.583

err:
The container(registrar) in pod(powermax-node-57lxq) is {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2025-01-22 20:44:26 +0000 UTC,FinishedAt:2025-01-22 20:44:56 +0000 UTC,ContainerID:containerd://dbbfe383b3deeb46d03c22690f62de87f9c474fa1283c0e5a89857f27e522732,}}
The container(registrar) in pod(powermax-node-7495z) is {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2025-01-22 20:44:27 +0000 UTC,FinishedAt:2025-01-22 20:44:57 +0000 UTC,ContainerID:containerd://facf847e66767621184a45fc77ac578306c33e0edb3097294786c4a1824a40f8,}}
  STEP:      Executing  Run custom test @ 01/22/25 20:45:31.732
  I0122 20:45:31.733145 2727 util.go:650] Running cert-csi [test vio --sc op-e2e-pmax --chainLength 1 --chainNumber 1]
[2025-01-22 20:45:31]  INFO Starting cert-csi; ver. 1.4.1
[2025-01-22 20:45:31]  INFO Using EVENT observer type
[2025-01-22 20:45:31]  INFO Using config from /root/.kube/config
[2025-01-22 20:45:31]  INFO Successfully loaded config. Host: https://10.227.248.175:6443
[2025-01-22 20:45:31]  INFO Created new KubeClient
[2025-01-22 20:45:31]  INFO Running 1 iteration(s)
[2025-01-22 20:45:31]  INFO     *** ITERATION NUMBER 1 ***
[2025-01-22 20:45:31]  INFO Starting VolumeIoSuite with op-e2e-pmax storage class
[2025-01-22 20:45:32]  INFO Successfully created namespace volumeio-test-465a8a65
[2025-01-22 20:45:32]  INFO Using default number of volumes
[2025-01-22 20:45:32]  INFO Using default image: docker.io/centos:latest
[2025-01-22 20:45:32]  INFO Creating IO pod
[2025-01-22 20:45:32]  INFO Waiting for pod iowriter-test-tzkn9 to be READY
[2025-01-22 20:46:16]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2025-01-22 20:46:17]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2025-01-22 20:46:19]  INFO Waiting until no Volume Attachments with PV  left
[2025-01-22 20:46:19]  INFO VolumeAttachment deleted
[2025-01-22 20:46:19]  INFO Deleting all resources in namespace volumeio-test-465a8a65
[2025-01-22 20:46:32]  INFO Namespace volumeio-test-465a8a65 was deleted in 12.096190004s
[2025-01-22 20:46:32]  INFO SUCCESS: VolumeIoSuite in 1m0.320028452s
[2025-01-22 20:46:32]  INFO Started generating reports...
Collecting metrics
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-22 20:46:32]  INFO Started generating reports...
Collecting metrics
Generating plots
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 35 p/s[2025-01-22 20:46:32]  WARN No ResourceUsageMetrics provided
[2025-01-22 20:46:32] ERROR no ResourceUsageMetrics provided
report-test-run-ef2e29bf:
Name: test-run-ef2e29bf
Host: https://10.227.248.175:6443
StorageClass: op-e2e-pmax
Minimum and Maximum EntityOverTime charts:

/root/.cert-csi/reports/test-run-ef2e29bf/PodsCreatingOverTime.png

/root/.cert-csi/reports/test-run-ef2e29bf/PodsReadyOverTime.png

/root/.cert-csi/reports/test-run-ef2e29bf/PodsTerminatingOverTime.png

/root/.cert-csi/reports/test-run-ef2e29bf/PvcsCreatingOverTime.png

/root/.cert-csi/reports/test-run-ef2e29bf/PvcsBoundOverTime.png

Tests:
--------------------------------------------------------------
1. TestCase: VolumeIoSuite
            Started:   2025-01-22 20:45:31.937437275 +0000 UTC
            Ended:     2025-01-22 20:46:32.293758739 +0000 UTC
            Result:    SUCCESS

            Stage metrics:
                    PVCAttachment:
                        Avg: 2.440633537s
                        Min: 2.440633537s
                        Max: 2.440633537s
                        Histogram:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCAttachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCAttachment_boxplot.png
                    PVCBind:
                        Avg: 2.797723763s
                        Min: 2.797723763s
                        Max: 2.797723763s
                        Histogram:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCBind.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCBind_boxplot.png
                    PVCCreation:
                        Avg: 5.493833726s
                        Min: 5.493833726s
                        Max: 5.493833726s
                        Histogram:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCCreation_boxplot.png
                    PVCDeletion:
                        Avg: 26.296747ms
                        Min: 26.296747ms
                        Max: 26.296747ms
                        Histogram:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCDeletion_boxplot.png
                    PVCUnattachment:
                        Avg: 1.769776514s
                        Min: 1.769776514s
                        Max: 1.769776514s
                        Histogram:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCUnattachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PVCUnattachment_boxplot.png
                    PodCreation:
                        Avg: 42.146426581s
                        Min: 42.146426581s
                        Max: 42.146426581s
                        Histogram:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PodCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PodCreation_boxplot.png
                    PodDeletion:
                        Avg: 1.471637297s
                        Min: 1.471637297s
                        Max: 1.471637297s
                        Histogram:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PodDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/PodDeletion_boxplot.png
                        EntityNumberOverTime:
        /root/.cert-csi/reports/test-run-ef2e29bf/VolumeIoSuite11/EntityNumberOverTime.png

[2025-01-22 20:46:32]  INFO Avg time of a run:   47.98s
[2025-01-22 20:46:32]  INFO Avg time of a del:   12.10s
[2025-01-22 20:46:32]  INFO Avg time of all:     60.31s
[2025-01-22 20:46:32]  INFO During this run 100.0% of suites succeeded
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/22/25 20:46:32.812
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 20:46:32.886
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/22/25 20:46:32.938

err: found the following pods: powermax-controller-6b78ff6667-9kb7z,powermax-controller-6b78ff6667-f4kkq,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxAuthSidecar] @ 01/22/25 20:47:23.022
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:47:23.112
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:47:23.185
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:47:23.213
  STEP: Ending: Install PowerMax Driver(Sidecar)
   @ 01/22/25 20:47:23.292
  STEP: Starting: Install PowerMax Driver with TLS (Sidecar)  @ 01/22/25 20:47:28.296
  STEP: Returning false here @ 01/22/25 20:47:28.296
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 20:47:28.296
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:47:28.319
  STEP:      Executing  Set up reverse proxy tls secret with SAN namespace [powermax] @ 01/22/25 20:47:28.984
Temporary directory created at: /tmp/tls-setup3027028339
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:47:30.06
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 20:47:30.419
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:47:30.927
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 20:47:31.367
  I0122 20:47:31.367862 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 20:47:31.724045 2727 builder.go:146] stderr: ""
  I0122 20:47:31.724115 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/22/25 20:47:31.724

err: expected custom resource status to be Succeeded. Got: Failed
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 20:48:21.807

err:
The container(registrar) in pod(powermax-node-dlv87) is {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2025-01-22 20:48:07 +0000 UTC,FinishedAt:2025-01-22 20:48:37 +0000 UTC,ContainerID:containerd://a7f8eebdd4076a47b4552c3fe95eaf6d3b31829f38cbf15b5fbd0e4bb4d6957c,}}
  STEP:      Executing  Run custom test @ 01/22/25 20:49:11.948
  I0122 20:49:11.948991 2727 util.go:650] Running cert-csi [test vio --sc op-e2e-pmax --chainLength 1 --chainNumber 1]
[2025-01-22 20:49:11]  INFO Starting cert-csi; ver. 1.4.1
[2025-01-22 20:49:11]  INFO Using EVENT observer type
[2025-01-22 20:49:11]  INFO Using config from /root/.kube/config
[2025-01-22 20:49:11]  INFO Successfully loaded config. Host: https://10.227.248.175:6443
[2025-01-22 20:49:12]  INFO Created new KubeClient
[2025-01-22 20:49:12]  INFO Running 1 iteration(s)
[2025-01-22 20:49:12]  INFO     *** ITERATION NUMBER 1 ***
[2025-01-22 20:49:12]  INFO Starting VolumeIoSuite with op-e2e-pmax storage class
[2025-01-22 20:49:12]  INFO Successfully created namespace volumeio-test-76d0fc31
[2025-01-22 20:49:12]  INFO Using default number of volumes
[2025-01-22 20:49:12]  INFO Using default image: docker.io/centos:latest
[2025-01-22 20:49:12]  INFO Creating IO pod
[2025-01-22 20:49:12]  INFO Waiting for pod iowriter-test-qhlkp to be READY
[2025-01-22 20:49:50]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2025-01-22 20:49:51]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2025-01-22 20:49:54]  INFO Waiting until no Volume Attachments with PV  left
[2025-01-22 20:49:54]  INFO VolumeAttachment deleted
[2025-01-22 20:49:54]  INFO Deleting all resources in namespace volumeio-test-76d0fc31
[2025-01-22 20:50:06]  INFO Namespace volumeio-test-76d0fc31 was deleted in 12.085933253s
[2025-01-22 20:50:07]  INFO SUCCESS: VolumeIoSuite in 55.153460372s
[2025-01-22 20:50:07]  INFO Started generating reports...
Collecting metrics
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-22 20:50:07]  INFO Started generating reports...
Collecting metrics
Generating plots
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 49 p/s[2025-01-22 20:50:07]  WARN No ResourceUsageMetrics provided
[2025-01-22 20:50:07] ERROR no ResourceUsageMetrics provided
report-test-run-57ab943a:
Name: test-run-57ab943a
Host: https://10.227.248.175:6443
StorageClass: op-e2e-pmax
Minimum and Maximum EntityOverTime charts:

/root/.cert-csi/reports/test-run-57ab943a/PodsCreatingOverTime.png

/root/.cert-csi/reports/test-run-57ab943a/PodsReadyOverTime.png

/root/.cert-csi/reports/test-run-57ab943a/PodsTerminatingOverTime.png

/root/.cert-csi/reports/test-run-57ab943a/PvcsCreatingOverTime.png

/root/.cert-csi/reports/test-run-57ab943a/PvcsBoundOverTime.png

Tests:
--------------------------------------------------------------
1. TestCase: VolumeIoSuite
            Started:   2025-01-22 20:49:12.147109701 +0000 UTC
            Ended:     2025-01-22 20:50:07.343350411 +0000 UTC
            Result:    SUCCESS

            Stage metrics:
                    PVCAttachment:
                        Avg: 2.491559641s
                        Min: 2.491559641s
                        Max: 2.491559641s
                        Histogram:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCAttachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCAttachment_boxplot.png
                    PVCBind:
                        Avg: 2.362051615s
                        Min: 2.362051615s
                        Max: 2.362051615s
                        Histogram:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCBind.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCBind_boxplot.png
                    PVCCreation:
                        Avg: 5.54806903s
                        Min: 5.54806903s
                        Max: 5.54806903s
                        Histogram:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCCreation_boxplot.png
                    PVCDeletion:
                        Avg: 89.229018ms
                        Min: 89.229018ms
                        Max: 89.229018ms
                        Histogram:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCDeletion_boxplot.png
                    PVCUnattachment:
                        Avg: 1.736068394s
                        Min: 1.736068394s
                        Max: 1.736068394s
                        Histogram:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCUnattachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PVCUnattachment_boxplot.png
                    PodCreation:
                        Avg: 37.716113309s
                        Min: 37.716113309s
                        Max: 37.716113309s
                        Histogram:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PodCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PodCreation_boxplot.png
                    PodDeletion:
                        Avg: 1.048816558s
                        Min: 1.048816558s
                        Max: 1.048816558s
                        Histogram:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PodDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/PodDeletion_boxplot.png
                        EntityNumberOverTime:
        /root/.cert-csi/reports/test-run-57ab943a/VolumeIoSuite12/EntityNumberOverTime.png

[2025-01-22 20:50:07]  INFO Avg time of a run:   41.90s
[2025-01-22 20:50:07]  INFO Avg time of a del:   12.09s
[2025-01-22 20:50:07]  INFO Avg time of all:     55.15s
[2025-01-22 20:50:07]  INFO During this run 100.0% of suites succeeded
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/22/25 20:50:07.858
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 20:50:07.931
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/22/25 20:50:07.982

err: found the following pods: powermax-controller-bff7c699d-9mpz7,powermax-controller-bff7c699d-xtp9x,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxAuthSidecar] @ 01/22/25 20:50:58.074
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:50:58.132
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:50:58.184
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/22/25 20:50:58.216
  STEP: Ending: Install PowerMax Driver with TLS (Sidecar)
   @ 01/22/25 20:50:58.298
  STEP: Starting: Install PowerMax Driver(With Observability)  @ 01/22/25 20:51:03.303
  STEP: Returning false here @ 01/22/25 20:51:03.303
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 20:51:03.303
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:51:03.326
  STEP:      Executing  Set up reverse proxy tls secret namespace [powermax] @ 01/22/25 20:51:04.019
revproxy-certs secret already exists, skipping creation.
csirevproxy-tls-secret already exists, skipping creation.
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:51:04.302
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 20:51:04.669
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 20:51:05.166
  I0122 20:51:05.167065 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 20:51:05.506105 2727 builder.go:146] stderr: ""
  I0122 20:51:05.506184 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/22/25 20:51:05.506

err: expected custom resource status to be Succeeded. Got: Failed
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 20:51:55.611

err:
The container(registrar) in pod(powermax-node-rjq5j) is {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2025-01-22 20:51:42 +0000 UTC,FinishedAt:2025-01-22 20:52:12 +0000 UTC,ContainerID:containerd://bc01d7518678451939fdd223e35120c11aa0d298c501d6587bb4972e079d3417,}}
  STEP:      Executing  Validate [observability] module from CR [1] is installed @ 01/22/25 20:52:45.762
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/22/25 20:52:55.932
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 20:52:56.007
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/22/25 20:52:56.061

err: found the following pods: powermax-controller-6b78ff6667-bnx4d,powermax-controller-6b78ff6667-zn4rg,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxAuthSidecar] @ 01/22/25 20:53:46.163
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:53:46.227
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:53:46.28
  STEP: Ending: Install PowerMax Driver(With Observability)
   @ 01/22/25 20:53:46.312
  STEP: Starting: Install PowerMax Driver (With Auth V1 module)  @ 01/22/25 20:53:51.313
  STEP: Returning false here @ 01/22/25 20:53:51.313
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 20:53:51.313
  STEP:      Executing  Create [authorization-proxy-server] prerequisites from CR [1] @ 01/22/25 20:53:51.335
=== Creating Authorization Proxy Server Prerequisites ===

Deleting all CSM from namespace: authorization
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 20:53:59.321
  I0122 20:53:59.321913 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=authorization apply --validate=true -f -'
  I0122 20:53:59.812192 2727 builder.go:146] stderr: ""
  I0122 20:53:59.812279 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/authorization created\nconfigmap/csm-config-params created\n"
  STEP:      Executing  Validate [authorization-proxy-server] module from CR [1] is installed @ 01/22/25 20:53:59.812

err: failed to check for AuthorizationProxyServer installation in default-source-cluster:
The pod(proxy-server-5459bd4d65-rszvj) is Pending
The pod(role-service-747c9d6db5-5qmsv) is Pending
The pod(storage-service-6c7955667c-m4k2r) is Pending
The pod(tenant-service-677bc6f84b-q288p) is Pending
  STEP:      Executing  Configure authorization-proxy-server for [powermax] for CR [1] @ 01/22/25 20:54:50.048
=== Configuring Authorization Proxy Server ===
Address: csm-authorization.com
=====Waiting for everything to be up and running, adding a sleep time of 120 seconds before creating the role, tenant and role binding===
=== Generating Admin Token ===
=== Writing Admin Token to Tmp File ===


=== Checking Storage ===

=== Checking Storage ===
 /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml storage list --insecure --addr csm-authorization.com:32658

=== Creating Storage ===

=== Storage ===
 /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml storage create --type powermax --endpoint https://10.230.24.220:8443 --system-id 000120001607 --user smc --password smc --array-insecure --insecure --addr csm-authorization.com:32658


=== Creating Tenant ===

=== Tenant ===
 /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml tenant create -n PancakeGroup --insecure --addr csm-authorization.com:32658
=== Checking Roles ===
 /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml role list --insecure --addr csm-authorization.com:32658


=== Creating Role ===

=== Role ===
 /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml role create --role=CSIGold=powermax=000120001607=SRP_1=100000000 --insecure --addr csm-authorization.com:32658


=== Creating RoleBinding ===

=== Binding Role ===
 /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml rolebinding create --tenant PancakeGroup --role CSIGold --insecure --addr csm-authorization.com:32658


=== Generating token ===

=== Token ===
 /usr/local/bin/karavictl --admin-token /tmp/adminToken.yaml generate token --tenant PancakeGroup --insecure --addr csm-authorization.com:32658 --access-token-expiration 2h0m0s


=== Applying token ===

=== Token Applied ===

  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:56:56.537
  STEP:      Executing  Set up secret with template [testfiles/powermax-templates/csm-authorization-config.json] name [karavi-authorization-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 20:56:57.197
  STEP:      Executing  Set up reverse proxy tls secret namespace [powermax] @ 01/22/25 20:56:57.677
revproxy-certs secret already exists, skipping creation.
csirevproxy-tls-secret already exists, skipping creation.
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:56:57.97
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config_auth.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxReverseProxy] @ 01/22/25 20:56:58.405
  STEP:      Executing  Apply custom resource [2] @ 01/22/25 20:56:58.87
  I0122 20:56:58.870590 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 20:56:59.231911 2727 builder.go:146] stderr: ""
  I0122 20:56:59.231969 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [2] @ 01/22/25 20:56:59.232
  STEP:      Executing  Validate [powermax] driver from CR [2] is installed @ 01/22/25 20:57:19.335
  STEP:      Executing  Validate [authorization] module from CR [2] is installed @ 01/22/25 20:57:39.394
map[com.dell.karavi-authorization-proxy:true deployment.kubernetes.io/revision:1]
map[com.dell.karavi-authorization-proxy:true deprecated.daemonset.template.generation:1]
  STEP:      Executing  Run custom test @ 01/22/25 20:57:49.505
  I0122 20:57:49.505261 2727 util.go:650] Running cert-csi [test vio --sc op-e2e-pmax  --chainLength 1 --chainNumber 1]
[2025-01-22 20:57:49]  INFO Starting cert-csi; ver. 1.4.1
[2025-01-22 20:57:49]  INFO Using EVENT observer type
[2025-01-22 20:57:49]  INFO Using config from /root/.kube/config
[2025-01-22 20:57:49]  INFO Successfully loaded config. Host: https://10.227.248.175:6443
[2025-01-22 20:57:49]  INFO Created new KubeClient
[2025-01-22 20:57:49]  INFO Running 1 iteration(s)
[2025-01-22 20:57:49]  INFO     *** ITERATION NUMBER 1 ***
[2025-01-22 20:57:49]  INFO Starting VolumeIoSuite with op-e2e-pmax storage class
[2025-01-22 20:57:49]  INFO Successfully created namespace volumeio-test-5229b32d
[2025-01-22 20:57:49]  INFO Using default number of volumes
[2025-01-22 20:57:49]  INFO Using default image: docker.io/centos:latest
[2025-01-22 20:57:49]  INFO Creating IO pod
[2025-01-22 20:57:49]  INFO Waiting for pod iowriter-test-m4f2f to be READY
[2025-01-22 20:58:33]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2025-01-22 20:58:35]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2025-01-22 20:58:37]  INFO Waiting until no Volume Attachments with PV  left
[2025-01-22 20:58:37]  INFO VolumeAttachment deleted
[2025-01-22 20:58:37]  INFO Deleting all resources in namespace volumeio-test-5229b32d
[2025-01-22 20:58:49]  INFO Namespace volumeio-test-5229b32d was deleted in 12.087061997s
[2025-01-22 20:58:54]  INFO SUCCESS: VolumeIoSuite in 1m5.204226639s
[2025-01-22 20:58:54]  INFO Started generating reports...
Collecting metrics
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-22 20:58:54]  INFO Started generating reports...
Collecting metrics
Generating plots
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 814 p/s[2025-01-22 20:58:55]  WARN No ResourceUsageMetrics provided
[2025-01-22 20:58:55] ERROR no ResourceUsageMetrics provided
report-test-run-a30f5d29:
Name: test-run-a30f5d29
Host: https://10.227.248.175:6443
StorageClass: op-e2e-pmax
Minimum and Maximum EntityOverTime charts:

/root/.cert-csi/reports/test-run-a30f5d29/PodsCreatingOverTime.png

/root/.cert-csi/reports/test-run-a30f5d29/PodsReadyOverTime.png

/root/.cert-csi/reports/test-run-a30f5d29/PodsTerminatingOverTime.png

/root/.cert-csi/reports/test-run-a30f5d29/PvcsCreatingOverTime.png

/root/.cert-csi/reports/test-run-a30f5d29/PvcsBoundOverTime.png

Tests:
--------------------------------------------------------------
1. TestCase: VolumeIoSuite
            Started:   2025-01-22 20:57:49.699523179 +0000 UTC
            Ended:     2025-01-22 20:58:54.946112642 +0000 UTC
            Result:    SUCCESS

            Stage metrics:
                    PVCAttachment:
                        Avg: 6.567609774s
                        Min: 6.567609774s
                        Max: 6.567609774s
                        Histogram:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCAttachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCAttachment_boxplot.png
                    PVCBind:
                        Avg: 2.801020569s
                        Min: 2.801020569s
                        Max: 2.801020569s
                        Histogram:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCBind.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCBind_boxplot.png
                    PVCCreation:
                        Avg: 9.673839485s
                        Min: 9.673839485s
                        Max: 9.673839485s
                        Histogram:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCCreation_boxplot.png
                    PVCDeletion:
                        Avg: 9.990115ms
                        Min: 9.990115ms
                        Max: 9.990115ms
                        Histogram:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCDeletion_boxplot.png
                    PVCUnattachment:
                        Avg: 1.859933055s
                        Min: 1.859933055s
                        Max: 1.859933055s
                        Histogram:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCUnattachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PVCUnattachment_boxplot.png
                    PodCreation:
                        Avg: 43.733969577s
                        Min: 43.733969577s
                        Max: 43.733969577s
                        Histogram:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PodCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PodCreation_boxplot.png
                    PodDeletion:
                        Avg: 1.994409295s
                        Min: 1.994409295s
                        Max: 1.994409295s
                        Histogram:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PodDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/PodDeletion_boxplot.png
                        EntityNumberOverTime:
        /root/.cert-csi/reports/test-run-a30f5d29/VolumeIoSuite13/EntityNumberOverTime.png

[2025-01-22 20:58:55]  INFO Avg time of a run:   48.02s
[2025-01-22 20:58:55]  INFO Avg time of a del:   12.09s
[2025-01-22 20:58:55]  INFO Avg time of all:     65.20s
[2025-01-22 20:58:55]  INFO During this run 100.0% of suites succeeded
  STEP:      Executing  Enable forceRemoveDriver on CR [2] @ 01/22/25 20:58:55.461
  STEP:      Executing  Delete custom resource [2] @ 01/22/25 20:58:55.548
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 20:58:55.602
  STEP:      Executing  Validate [powermax] driver from CR [2] is not installed @ 01/22/25 20:58:55.647

err: found the following pods: powermax-controller-5d54ffb8d9-gncqs,powermax-controller-5d54ffb8d9-lmwc2,
  STEP:      Executing  Restore template [testfiles/powermax-templates/csm-authorization-config.json] for [pmaxAuthSidecar] @ 01/22/25 20:59:45.737
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:59:45.785
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config_auth.yaml] for [pmaxReverseProxy] @ 01/22/25 20:59:45.815
  STEP: Ending: Install PowerMax Driver (With Auth V1 module)
   @ 01/22/25 20:59:45.842
  STEP: Starting: Install Powermax Driver(Standalone), Enable Resiliency  @ 01/22/25 20:59:50.842
  STEP: Returning false here @ 01/22/25 20:59:50.843
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 20:59:50.843
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 20:59:50.864
  STEP:      Executing  Set up reverse proxy tls secret namespace [powermax] @ 01/22/25 20:59:51.517
revproxy-certs secret already exists, skipping creation.
csirevproxy-tls-secret already exists, skipping creation.
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 20:59:51.794
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 20:59:52.191
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 20:59:52.664
  I0122 20:59:52.665173 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 20:59:53.029266 2727 builder.go:146] stderr: ""
  I0122 20:59:53.029320 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/22/25 20:59:53.029
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 21:00:13.089
  STEP:      Executing  Validate [resiliency] module from CR [1] is not installed @ 01/22/25 21:00:33.165
  STEP:      Executing  Enable [resiliency] module from CR [1] @ 01/22/25 21:00:43.246
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 21:00:58.332

err:
The pod(powermax-controller-67bf5977b5-6lmfq) is Pending
  STEP:      Executing  Validate [resiliency] module from CR [1] is installed @ 01/22/25 21:01:48.465
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/22/25 21:01:58.566
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 21:01:58.634
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/22/25 21:01:58.686

err: found the following pods: csipowermax-reverseproxy-bbbc6c6b7-jkvdh,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxAuthSidecar] @ 01/22/25 21:02:48.748
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 21:02:48.809
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 21:02:48.872
  STEP: Ending: Install Powermax Driver(Standalone), Enable Resiliency
   @ 01/22/25 21:02:48.907
  STEP: Starting: Install Powermax Driver(With Resiliency), Disable Resiliency module  @ 01/22/25 21:02:53.908
  STEP: Returning false here @ 01/22/25 21:02:53.908
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/22/25 21:02:53.908
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 21:02:53.929
  STEP:      Executing  Set up reverse proxy tls secret namespace [powermax] @ 01/22/25 21:02:54.641
revproxy-certs secret already exists, skipping creation.
csirevproxy-tls-secret already exists, skipping creation.
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 21:02:54.908
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] name [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 01/22/25 21:02:55.294
  STEP:      Executing  Apply custom resource [1] @ 01/22/25 21:02:55.794
  I0122 21:02:55.795058 2727 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0122 21:02:56.148859 2727 builder.go:146] stderr: ""
  I0122 21:02:56.148911 2727 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/22/25 21:02:56.148

err: expected custom resource status to be Succeeded. Got: Failed
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 21:03:46.256
  STEP:      Executing  Validate [resiliency] module from CR [1] is installed @ 01/22/25 21:04:06.315
  STEP:      Executing  Disable [resiliency] module from CR [1] @ 01/22/25 21:04:16.429
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/22/25 21:04:16.502

err:
The pod(powermax-controller-6b78ff6667-5rm7p) is Pending
The pod(powermax-controller-6b78ff6667-wk2t9) is Pending

err:
The pod(powermax-controller-6b78ff6667-5rm7p) is Pending
The container(attacher) in pod(powermax-controller-6b78ff6667-wk2t9) is {&ContainerStateWaiting{Reason:CrashLoopBackOff,Message:back-off 10s restarting failed container=attacher pod=powermax-controller-6b78ff6667-wk2t9_powermax(4d5021a9-6426-47ce-83f3-e717d57b2d06),} nil nil}

err:
The container(attacher) in pod(powermax-controller-6b78ff6667-5rm7p) is {nil nil &ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:Lost connection to CSI driver, exiting,StartedAt:2025-01-22 21:05:26 +0000 UTC,FinishedAt:2025-01-22 21:05:32 +0000 UTC,ContainerID:containerd://18a8dd9bd2f9ae686d6edd38b00427314fa6c055b9fcbbe851aa3850c43cb5e5,}}
  STEP:      Executing  Validate [resiliency] module from CR [1] is not installed @ 01/22/25 21:06:06.782
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/22/25 21:06:16.865
  STEP:      Executing  Delete custom resource [1] @ 01/22/25 21:06:16.917
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/22/25 21:06:16.971

err: found the following pods: powermax-controller-6b78ff6667-5rm7p,powermax-controller-6b78ff6667-wk2t9,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxAuthSidecar] @ 01/22/25 21:07:07.033
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/22/25 21:07:07.085
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 01/22/25 21:07:07.129
  STEP: Ending: Install Powermax Driver(With Resiliency), Disable Resiliency module
   @ 01/22/25 21:07:07.156
• [1724.699 seconds]
------------------------------

Ran 1 of 1 Specs in 1725.272 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 28m56.154618859s
Test Suite Passed
  • Add Minimal Test for Powermax Use Secret
-> # bash run-e2e-test.sh --pmax
/root/git/public/dell/csm-operator/tests/e2e
  W0124 16:16:31.528517   14742 test_context.go:541] Unable to find in-cluster config, using default host : https://127.0.0.1:6443
  I0124 16:16:31.531926 14742 test_context.go:564] The --provider flag is not set. Continuing as if --provider=skeleton had been used.
Running Suite: CSM Operator End-to-End Tests - /root/git/public/dell/csm-operator/tests/e2e
===========================================================================================
Random Seed: 1737735377

Will run 1 of 1 specs
------------------------------
[BeforeSuite]
/root/git/public/dell/csm-operator/tests/e2e/e2e_test.go:98
  STEP: Getting test environment variables @ 01/24/25 16:16:31.534
  STEP: [powermax] @ 01/24/25 16:16:31.534
  STEP: Reading values file @ 01/24/25 16:16:31.534
  STEP: Getting a k8s client @ 01/24/25 16:16:31.549
[BeforeSuite] PASSED [0.029 seconds]
------------------------------
[run-e2e-test] E2E Testing Running all test Given Test Scenarios
/root/git/public/dell/csm-operator/tests/e2e/e2e_test.go:139
  STEP: Starting: Install PowerMax Driver with Mount Credentials (Minimal)  @ 01/24/25 16:16:31.563
  STEP: Returning false here @ 01/24/25 16:16:31.563
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 01/24/25 16:16:31.563
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/24/25 16:16:31.651
  STEP:      Executing  Set up secret with template [testfiles/powermax-templates/powermax-use-secret-template.yaml] name [powermax-creds] in namespace [powermax] for [pmaxUseSecret] @ 01/24/25 16:16:33.027
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/24/25 16:16:33.804
  STEP:      Executing  Apply custom resource [1] @ 01/24/25 16:16:34.596
  I0124 16:16:34.596821 14742 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I0124 16:16:35.154799 14742 builder.go:146] stderr: ""
  I0124 16:16:35.154864 14742 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 01/24/25 16:16:35.154

err: expected custom resource status to be Succeeded. Got: Failed
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 01/24/25 16:17:25.254
  STEP:      Executing  Validate [powermax] driver spec from CR [1] @ 01/24/25 16:17:45.334
  STEP:      Executing  Run custom test @ 01/24/25 16:17:45.356
  I0124 16:17:45.356123 14742 util.go:650] Running cert-csi [test vio --sc op-e2e-pmax --chainLength 1 --chainNumber 1]
[2025-01-24 16:17:45]  INFO Starting cert-csi; ver. 1.4.1
[2025-01-24 16:17:45]  INFO Using EVENT observer type
[2025-01-24 16:17:45]  INFO Using config from /root/.kube/config
[2025-01-24 16:17:45]  INFO Successfully loaded config. Host: https://10.227.248.175:6443
[2025-01-24 16:17:45]  INFO Created new KubeClient
[2025-01-24 16:17:45]  INFO Running 1 iteration(s)
[2025-01-24 16:17:45]  INFO     *** ITERATION NUMBER 1 ***
[2025-01-24 16:17:45]  INFO Starting VolumeIoSuite with op-e2e-pmax storage class
[2025-01-24 16:17:45]  INFO Successfully created namespace volumeio-test-e3df4ab4
[2025-01-24 16:17:45]  INFO Using default number of volumes
[2025-01-24 16:17:45]  INFO Using default image: docker.io/centos:latest
[2025-01-24 16:17:45]  INFO Creating IO pod
[2025-01-24 16:17:45]  INFO Waiting for pod iowriter-test-x65d8 to be READY
[2025-01-24 16:18:27]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2025-01-24 16:18:28]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2025-01-24 16:18:31]  INFO Waiting until no Volume Attachments with PV  left
[2025-01-24 16:18:31]  INFO VolumeAttachment deleted
[2025-01-24 16:18:31]  INFO Deleting all resources in namespace volumeio-test-e3df4ab4
[2025-01-24 16:18:43]  INFO Namespace volumeio-test-e3df4ab4 was deleted in 12.089884509s
[2025-01-24 16:18:45]  INFO SUCCESS: VolumeIoSuite in 1m0.178780255s
[2025-01-24 16:18:45]  INFO Started generating reports...
Collecting metrics
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-24 16:18:45]  INFO Started generating reports...
Collecting metrics
Generating plots
1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2025-01-24 16:18:46]  WARN No ResourceUsageMetrics provided
[2025-01-24 16:18:46] ERROR no ResourceUsageMetrics provided
report-test-run-d51e5efc:
Name: test-run-d51e5efc
Host: https://10.227.248.175:6443
StorageClass: op-e2e-pmax
Minimum and Maximum EntityOverTime charts:

/root/.cert-csi/reports/test-run-d51e5efc/PodsCreatingOverTime.png

/root/.cert-csi/reports/test-run-d51e5efc/PodsReadyOverTime.png

/root/.cert-csi/reports/test-run-d51e5efc/PodsTerminatingOverTime.png

/root/.cert-csi/reports/test-run-d51e5efc/PvcsCreatingOverTime.png

/root/.cert-csi/reports/test-run-d51e5efc/PvcsBoundOverTime.png

Tests:
--------------------------------------------------------------
1. TestCase: VolumeIoSuite
            Started:   2025-01-24 16:17:45.56952127 +0000 UTC
            Ended:     2025-01-24 16:18:45.790783049 +0000 UTC
            Result:    SUCCESS

            Stage metrics:
                    PVCAttachment:
                        Avg: 6.300474034s
                        Min: 6.300474034s
                        Max: 6.300474034s
                        Histogram:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCAttachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCAttachment_boxplot.png
                    PVCBind:
                        Avg: 2.803956782s
                        Min: 2.803956782s
                        Max: 2.803956782s
                        Histogram:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCBind.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCBind_boxplot.png
                    PVCCreation:
                        Avg: 9.329152694s
                        Min: 9.329152694s
                        Max: 9.329152694s
                        Histogram:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCCreation_boxplot.png
                    PVCDeletion:
                        Avg: 11.188354ms
                        Min: 11.188354ms
                        Max: 11.188354ms
                        Histogram:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCDeletion_boxplot.png
                    PVCUnattachment:
                        Avg: 1.875946326s
                        Min: 1.875946326s
                        Max: 1.875946326s
                        Histogram:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCUnattachment.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PVCUnattachment_boxplot.png
                    PodCreation:
                        Avg: 41.148407127s
                        Min: 41.148407127s
                        Max: 41.148407127s
                        Histogram:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PodCreation.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PodCreation_boxplot.png
                    PodDeletion:
                        Avg: 1.368655756s
                        Min: 1.368655756s
                        Max: 1.368655756s
                        Histogram:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PodDeletion.png
                        BoxPlot:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/PodDeletion_boxplot.png
                        EntityNumberOverTime:
        /root/.cert-csi/reports/test-run-d51e5efc/VolumeIoSuite15/EntityNumberOverTime.png

[2025-01-24 16:18:46]  INFO Avg time of a run:   46.05s
[2025-01-24 16:18:46]  INFO Avg time of a del:   12.09s
[2025-01-24 16:18:46]  INFO Avg time of all:     60.17s
[2025-01-24 16:18:46]  INFO During this run 100.0% of suites succeeded
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 01/24/25 16:18:46.365
  STEP:      Executing  Delete custom resource [1] @ 01/24/25 16:18:46.414
  STEP:      Executing  Validate [powermax] driver from CR [1] is not installed @ 01/24/25 16:18:46.462

err: found the following pods: powermax-controller-76f78b4dc9-5z2c6,powermax-controller-76f78b4dc9-grq9v,
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 01/24/25 16:19:36.561
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-use-secret-template.yaml] for [pmaxUseSecret] @ 01/24/25 16:19:36.654
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-array-config.yaml] for [pmaxArrayConfig] @ 01/24/25 16:19:36.77
  STEP: Ending: Install PowerMax Driver with Mount Credentials (Minimal)
   @ 01/24/25 16:19:36.912
• [190.353 seconds]
------------------------------

Ran 1 of 1 Specs in 190.382 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 3m24.791727409s
Test Suite Passed

@falfaroc falfaroc marked this pull request as ready for review January 24, 2025 16:33
lukeatdell
lukeatdell reviewed Jan 24, 2025
View reviewed changes
@falfaroc falfaroc merged commit d604df3 into usr/spark/mount-credentials Jan 27, 2025
7 checks passed
@falfaroc falfaroc deleted the usr/falfaroc/powermax-support-minimal-manifest branch January 27, 2025 20:27
falfaroc added a commit that referenced this pull request Feb 14, 2025
@falfaroc
Update Powermax to handle Minimal Manifests with Mount Credentials (#872
1bea78c 
)

* Support Powermax mount credentials with minimal manifest

* Fix Powermax reverse proxy minimal manifest confusion

* Fix Powermax unit tests

* Fix linting issues

* Add PowerMax Mount Credentials minimal manifest e2e test

* Fix warning syntax
@falfaroc falfaroc mentioned this pull request Feb 14, 2025
Merged
12 tasks
xuluna pushed a commit that referenced this pull request Feb 26, 2025
@falfaroc @xuluna
Update Powermax to handle Minimal Manifests with Mount Credentials (#872
77e6d1f 
)

* Support Powermax mount credentials with minimal manifest

* Fix Powermax reverse proxy minimal manifest confusion

* Fix Powermax unit tests

* Fix linting issues

* Add PowerMax Mount Credentials minimal manifest e2e test

* Fix warning syntax
xuluna pushed a commit that referenced this pull request Feb 27, 2025
@falfaroc @xuluna
Update Powermax to handle Minimal Manifests with Mount Credentials (#872
47add0a 
)

* Support Powermax mount credentials with minimal manifest

* Fix Powermax reverse proxy minimal manifest confusion

* Fix Powermax unit tests

* Fix linting issues

* Add PowerMax Mount Credentials minimal manifest e2e test

* Fix warning syntax
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bharathsreekanth bharathsreekanth bharathsreekanth approved these changes

@lukeatdell lukeatdell lukeatdell approved these changes

@abhi16394 abhi16394 Awaiting requested review from abhi16394

@mbasha-dell mbasha-dell Awaiting requested review from mbasha-dell

@alikdell alikdell Awaiting requested review from alikdell

@chimanjain chimanjain Awaiting requested review from chimanjain

@coulof coulof Awaiting requested review from coulof

@Deepak-Ghivari Deepak-Ghivari Awaiting requested review from Deepak-Ghivari

@gallacher gallacher Awaiting requested review from gallacher

@gilltaran gilltaran Awaiting requested review from gilltaran

@harishp8889 harishp8889 Awaiting requested review from harishp8889

@HarishH-DELL HarishH-DELL Awaiting requested review from HarishH-DELL

@JacobGros JacobGros Awaiting requested review from JacobGros

@jooseppi-luna jooseppi-luna Awaiting requested review from jooseppi-luna

@karthikk92 karthikk92 Awaiting requested review from karthikk92

@kumarkgosa kumarkgosa Awaiting requested review from kumarkgosa

@bandak2 bandak2 Awaiting requested review from bandak2

@meggm meggm Awaiting requested review from meggm

@mgandharva mgandharva Awaiting requested review from mgandharva

@mjsdell mjsdell Awaiting requested review from mjsdell

@nitesh3108 nitesh3108 Awaiting requested review from nitesh3108

@Prabhu-Dell Prabhu-Dell Awaiting requested review from Prabhu-Dell

@rajendraindukuri rajendraindukuri Awaiting requested review from rajendraindukuri

@rajkumar-palani rajkumar-palani Awaiting requested review from rajkumar-palani

@shefali-malhotra shefali-malhotra Awaiting requested review from shefali-malhotra

@panigs7 panigs7 Awaiting requested review from panigs7

@tdawe tdawe Awaiting requested review from tdawe

@shaynafinocchiaro shaynafinocchiaro Awaiting requested review from shaynafinocchiaro

@atye atye Awaiting requested review from atye

@sharmilarama sharmilarama Awaiting requested review from sharmilarama

@shanmydell shanmydell Awaiting requested review from shanmydell

@xuluna xuluna Awaiting requested review from xuluna

@harshitap26 harshitap26 Awaiting requested review from harshitap26

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@falfaroc @bharathsreekanth @lukeatdell