[BUG]: Powerstore Multiple iSCSI network support for CSI driver

[AkshaySainiDell]

Bug Description

Customer needs to be able to address multiple iSCSI networks with the driver. As it stands, the driver is only able to address one iSCSI network.
Following is referenced:
Container Storage Modules | Dell Technologies
Dell PowerStore CSI Driver 1.3 for Kubernetes and RedHat OpenShift – What’s New? - Itzikr's Blog (volumes.blog)
Dell Container Storage Modules (CSM) v1.3, Are now available - Itzikr's Blog (volumes.blog)

Customers Environment:
Our architecture looks like this:

As you can see, there are different network connections for accessing the PowerStore appliances depending on the "consumer". The networks are completely separated from each other.
Our Kubernetes nodes easily reach the IP addresses from the network 192.168.176.64/26.
The installation of the CSI driver was carried out according to https://dell.github.io/csm-docs/docs/csidriver/installation/helm/powerstore/ [dell.github.io] controllers, and node pods were launched. All Kubernetes nodes are registered on the PowerStore system.
But that was it. It is not possible to define volumes by PVC, the PVC remain in the "Pending" state.
If you take a closer look at this, you will see that none of the Kubernetes nodes can establish an iSCSI connection to the storage system.
In the Node-Driver Logs (see below) there are indications that the driver only tries to establish an iSCSI connection to one address per storage system – and for the first one. The cause lies in https://github.com/dell/csi-powerstore/blob/72b2d20fa739cb5ea9176fe0faaeab122d3d485d/pkg/node/node.go#L1104 [github.com]
There, the FIRST of all available portal IP addresses is simply used to perform a discovery – the driver tacitly assumes that he can reach this address.
However, this does not work for us due to the network separation outlined above. The 10.21.0.x addresses are not reachable (and this is intentional). And even if you were reachable – what if this is not the best possible path? Why can't you just specify the iSCSI addresses to be used for discovery when defining the "Conenction Information" to the PowerStore array?
And something else – the support for "Kubernetes Topology Support" is also rather sparse.
„This Topology support does not include customer-defined topology, users cannot create their own labels for nodes, they must use whatever labels are returned by the driver and applied automatically by Kubernetes on its nodes.“
Unfortunately, it is not possible for us to distribute workloads to storage systems in different zones/regions. This is easily implemented by other providers (e.g. PureStorage, NetApp, etc.).

Logs

{"level":"debug","msg":"REQUEST: GET /api/rest/ip_pool_address?order=id\u0026purposes=cs.%7BStorage_Iscsi_Target%7D\u0026select=address%2Cappliance_id%2Cid%2Cip_port_id%2Cip_port%28target_iqn%2C+id%29%2Cnetwork_id%2Cnode_id%2Cpurposes HTTP/1.1 Host: 10.60.27.12 Application-Type: CSI Driver for Dell EMC PowerStore/2.4.0 Authorization: ****** ","time":"2022-12-12T08:54:07.858570747Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 1116 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff [{"address":"10.21.0.13","appliance_id":"A1","id":"IP18","ip_port_id":"IP_PORT8","network_id":"NW6","node_id":"N2","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-b-526000c7","id":"IP_PORT8"}}, {"address":"10.21.0.14","appliance_id":"A1","id":"IP19","ip_port_id":"IP_PORT4","network_id":"NW6","node_id":"N1","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-a-212fa3f1","id":"IP_PORT4"}}, {"address":"192.168.176.69","appliance_id":"A1","id":"IP27","ip_port_id":"IP_PORT4","network_id":"NW9","node_id":"N1","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-a-212fa3f1","id":"IP_PORT4"}}, {"address":"192.168.176.70","appliance_id":"A1","id":"IP28","ip_port_id":"IP_PORT8","network_id":"NW9","node_id":"N2","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-b-526000c7","id":"IP_PORT8"}}]\n","time":"2022-12-12T08:54:07.871344367Z"}
{"level":"debug","msg":"release a lock","time":"2022-12-12T08:54:07.871501772Z"}
{"level":"error","msg":"couldn't discover targets","time":"2022-12-12T08:55:42.889865867Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/ip_pool_address?order=id\u0026purposes=cs.%7BStorage_Iscsi_Target%7D\u0026select=address%2Cappliance_id%2Cid%2Cip_port_id%2Cip_port%28target_iqn%2C+id%29%2Cnetwork_id%2Cnode_id%2Cpurposes HTTP/1.1 Host: 10.61.27.12 Application-Type: CSI Driver for Dell EMC PowerStore/2.4.0 Authorization: ****** ","time":"2022-12-12T08:55:42.890253514Z"}
{"level":"debug","msg":"acquire a lock","time":"2022-12-12T08:55:42.890281898Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 1118 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff [{"address":"10.21.0.16","appliance_id":"A1","id":"IP18","ip_port_id":"IP_PORT4","network_id":"NW6","node_id":"N1","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-a-441a8e01","id":"IP_PORT4"}}, {"address":"10.21.0.20","appliance_id":"A1","id":"IP19","ip_port_id":"IP_PORT8","network_id":"NW6","node_id":"N2","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-b-173a1ac4","id":"IP_PORT8"}}, {"address":"192.168.176.133","appliance_id":"A1","id":"IP21","ip_port_id":"IP_PORT8","network_id":"NW7","node_id":"N2","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-b-173a1ac4","id":"IP_PORT8"}}, {"address":"192.168.176.134","appliance_id":"A1","id":"IP22","ip_port_id":"IP_PORT4","network_id":"NW7","node_id":"N1","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-a-441a8e01","id":"IP_PORT4"}}]\n","time":"2022-12-12T08:55:42.909870839Z"}
{"level":"debug","msg":"release a lock","time":"2022-12-12T08:55:42.90994131Z"}
{"level":"info","msg":"/csi.v1.Identity/GetPluginInfo: REQ 0003: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:56:09.54592005Z"}
{"level":"info","msg":"/csi.v1.Identity/GetPluginInfo: REP 0003: Name=csi-powerstore.dellemc.com, VendorVersion=2.4.0, Manifest=map[commit:bdb868e360b2a72ab58a0c766e7760031cc75721 formed:Wed, 21 Sep 2022 09:21:25 UTC semver:2.4.0 url:https://github.com/dell/csi-powerstore [github.com]], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:56:09.546021677Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetInfo: REQ 0004: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:56:10.93849792Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/ip_pool_address?order=id\u0026purposes=cs.%7BStorage_Iscsi_Target%7D\u0026select=address%2Cappliance_id%2Cid%2Cip_port_id%2Cip_port%28target_iqn%2C+id%29%2Cnetwork_id%2Cnode_id%2Cpurposes HTTP/1.1 Host: 10.60.27.12 Application-Type: CSI Driver for Dell EMC PowerStore/2.4.0 Authorization: ****** ","time":"2022-12-12T08:56:10.939137008Z"}
{"level":"debug","msg":"acquire a lock","time":"2022-12-12T08:56:10.939205016Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 1116 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff [{"address":"10.21.0.13","appliance_id":"A1","id":"IP18","ip_port_id":"IP_PORT8","network_id":"NW6","node_id":"N2","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-b-526000c7","id":"IP_PORT8"}}, {"address":"10.21.0.14","appliance_id":"A1","id":"IP19","ip_port_id":"IP_PORT4","network_id":"NW6","node_id":"N1","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-a-212fa3f1","id":"IP_PORT4"}}, {"address":"192.168.176.69","appliance_id":"A1","id":"IP27","ip_port_id":"IP_PORT4","network_id":"NW9","node_id":"N1","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-a-212fa3f1","id":"IP_PORT4"}}, {"address":"192.168.176.70","appliance_id":"A1","id":"IP28","ip_port_id":"IP_PORT8","network_id":"NW9","node_id":"N2","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-b-526000c7","id":"IP_PORT8"}}]\n","time":"2022-12-12T08:56:10.958444176Z"}
{"level":"debug","msg":"release a lock","time":"2022-12-12T08:56:10.958643968Z"}
Error discovering 10.21.0.13:3260: exit status 4
{"level":"error","msg":"couldn't discover targets","time":"2022-12-12T08:57:17.927256833Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetInfo: REP 0002: NodeId=k8s01-csi-node-29817fed32084f169eef5ff1c423189a-192.168.128.17, MaxVolumesPerNode=0, AccessibleTopology=segments:\u003ckey:"csi-powerstore.dellemc.com/10.60.27.12-nfs" value:"true" \u003e segments:\u003ckey:"csi-powerstore.dellemc.com/10.61.27.12-nfs" value:"true" \u003e , XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:57:17.927425096Z"}
{"level":"error","msg":"couldn't discover targets","time":"2022-12-12T08:57:45.974360928Z"}
Error discovering 10.21.0.16:3260: exit status 4
{"level":"debug","msg":"REQUEST: GET /api/rest/ip_pool_address?order=id\u0026purposes=cs.%7BStorage_Iscsi_Target%7D\u0026select=address%2Cappliance_id%2Cid%2Cip_port_id%2Cip_port%28target_iqn%2C+id%29%2Cnetwork_id%2Cnode_id%2Cpurposes HTTP/1.1 Host: 10.61.27.12 Application-Type: CSI Driver for Dell EMC PowerStore/2.4.0 Authorization: ****** ","time":"2022-12-12T08:57:45.974699415Z"}
{"level":"debug","msg":"acquire a lock","time":"2022-12-12T08:57:45.974738096Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 1118 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff [{"address":"10.21.0.16","appliance_id":"A1","id":"IP18","ip_port_id":"IP_PORT4","network_id":"NW6","node_id":"N1","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-a-441a8e01","id":"IP_PORT4"}}, {"address":"10.21.0.20","appliance_id":"A1","id":"IP19","ip_port_id":"IP_PORT8","network_id":"NW6","node_id":"N2","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-b-173a1ac4","id":"IP_PORT8"}}, {"address":"192.168.176.133","appliance_id":"A1","id":"IP21","ip_port_id":"IP_PORT8","network_id":"NW7","node_id":"N2","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-b-173a1ac4","id":"IP_PORT8"}}, {"address":"192.168.176.134","appliance_id":"A1","id":"IP22","ip_port_id":"IP_PORT4","network_id":"NW7","node_id":"N1","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214119-a-441a8e01","id":"IP_PORT4"}}]\n","time":"2022-12-12T08:57:45.989144089Z"}
{"level":"debug","msg":"release a lock","time":"2022-12-12T08:57:45.989330716Z"}
{"level":"info","msg":"/csi.v1.Identity/GetPluginInfo: REQ 0005: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:58:24.044202533Z"}
{"level":"info","msg":"/csi.v1.Identity/GetPluginInfo: REP 0005: Name=csi-powerstore.dellemc.com, VendorVersion=2.4.0, Manifest=map[commit:bdb868e360b2a72ab58a0c766e7760031cc75721 formed:Wed, 21 Sep 2022 09:21:25 UTC semver:2.4.0 url:https://github.com/dell/csi-powerstore [github.com]], XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:58:24.044396166Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetInfo: REQ 0006: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:58:26.022487322Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/ip_pool_address?order=id\u0026purposes=cs.%7BStorage_Iscsi_Target%7D\u0026select=address%2Cappliance_id%2Cid%2Cip_port_id%2Cip_port%28target_iqn%2C+id%29%2Cnetwork_id%2Cnode_id%2Cpurposes HTTP/1.1 Host: 10.60.27.12 Application-Type: CSI Driver for Dell EMC PowerStore/2.4.0 Authorization: ****** ","time":"2022-12-12T08:58:26.022928301Z"}
{"level":"debug","msg":"acquire a lock","time":"2022-12-12T08:58:26.022977766Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 1116 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=******; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff [{"address":"10.21.0.13","appliance_id":"A1","id":"IP18","ip_port_id":"IP_PORT8","network_id":"NW6","node_id":"N2","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-b-526000c7","id":"IP_PORT8"}}, {"address":"10.21.0.14","appliance_id":"A1","id":"IP19","ip_port_id":"IP_PORT4","network_id":"NW6","node_id":"N1","purposes":["Storage_Iscsi_Target","External_Replication_Iscsi"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-a-212fa3f1","id":"IP_PORT4"}}, {"address":"192.168.176.69","appliance_id":"A1","id":"IP27","ip_port_id":"IP_PORT4","network_id":"NW9","node_id":"N1","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-a-212fa3f1","id":"IP_PORT4"}}, {"address":"192.168.176.70","appliance_id":"A1","id":"IP28","ip_port_id":"IP_PORT8","network_id":"NW9","node_id":"N2","purposes":["Storage_Iscsi_Target"],"ip_port":{"target_iqn":"iqn.2015-10.com.dell:dellemc-powerstore-crk00224214120-b-526000c7","id":"IP_PORT8"}}]\n","time":"2022-12-12T08:58:26.03736463Z"}
{"level":"debug","msg":"release a lock","time":"2022-12-12T08:58:26.037508509Z"}
{"level":"error","msg":"couldn't discover targets","time":"2022-12-12T08:59:21.010931295Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetInfo: REP 0004: NodeId=k8s01-csi-node-29817fed32084f169eef5ff1c423189a-192.168.128.17, MaxVolumesPerNode=0, AccessibleTopology=segments:\u003ckey:"csi-powerstore.dellemc.com/10.60.27.12-nfs" value:"true" \u003e segments:\u003ckey:"csi-powerstore.dellemc.com/10.61.27.12-nfs" value:"true" \u003e , XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2022-12-12T08:59:21.011059326Z"}
Error discovering 10.21.0.13:3260: exit status 4
Error discovering 10.21.0.16:3260: exit status 4

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

Configure multiple iSCSI networks on the array and install the driver.

Expected Behavior

Driver should address multiple iSCSI networks.

CSM Driver(s)

CSI Driver for PowerStore v2.5

Installation Type

No response

Container Storage Modules Enabled

No response

Container Orchestrator

Kubernetes 1.25

Operating System

CentOS Linux release 7.7.1908 (Core)

[AkshaySainiDell]

The issue has been fixed and will be released in CSM 1.6.