[BUG]: Multi Controller defect - sidecars timeout

[adamginna-dell]

Bug Description

The first controller pods start without any problems. Second and next controller pods are failing.
Link to Change in a csi-lib-utils library that add timeout for connection with socket. New versions of sidecars use new version of csi-lib-utils and timeout cause controller pods fail.

Logs

level=info msg="Attempting driver connection at: unix:/var/run/csi/csi.sock"
level=debug msg="grpc.Dial returned context deadline exceeded"
level=error msg="Waiting on connection to driver csi.sock: context deadline exceeded"
level=debug msg="grpc.Dial returned context deadline exceeded"
level=error msg="Waiting on connection to driver csi.sock: context deadline exceeded"

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

Setup controller replicas in to number more then 1.

Expected Behavior

All controller pods are in Running state.

CSM Driver(s)

CSI Driver - Unity 2.9.0, Powerscale 2.9.0, Powerflex 2.9.0, Powermax 2.9.0

Installation Type

No response

Container Storage Modules Enabled

No response

Container Orchestrator

All

Operating System

All

[adamginna-dell]

Standby controller pod is in crashloopbackoff state. As a workaround scale down the replica count of the controller pod’s deployment to 1 using
kubectl scale deployment <deployment_name> --replicas=1 -n <driver_namespace>

[gallacher]

link: 20348