VNX: alu_hlu cache having entries for deleted luns

[Murray-LIANG]

OpenStack VNX tempest CI always failed. This issue was not always reproducible.
When the CI failed, you could see below log keeping printing:

Searching for a device in session 2 and hctl ('4', '0', '0', 101) yield: None

Cinder/Nova host cannot find the device because it even wasn't attached:

Mar 23 02:13:32 vnxvm1 cinder-volume[27333]: INFO storops.vnx.resource.sg [None req-9f5f1115-5f0c-4f69-afab-f03c1e76a76c tempest-Volume
RetypeWithMigrationTest-934831485 None] ALU(alu=29) is already attached, found in the cache(hlu=101).

The ALU found in the cache will not be attached on VNX. So, the problem is the alu-hlu cache is dirty.

When there are more than two pools in OpenStack VNX Cinder, there will be more than backends running, one per pool. In each running backend, there is an sg cache. And inside the sg cache, there is an alu-hlu-mapping cache.

This issue occurred in the below scenario:

1. Backend-A's sg cache had the entry for LUN-A's alu.
2. Backend-B's sg cache had the entry for LUN-A's alu too.
3. Backend-B detached and deleted LUN-A.
4. Backend-A created a new LUN-B but with the same alu with LUN-A.
5. Backend-A tried to attach LUN-B and found it was already in the cache. So, the cache was dirty.

[Murray-LIANG]

+------------------+      +-----------------+
|Cinder Backend-A  |      |Cinder Backend-B |
|                  |      |                 |
|    sg cache      |      |    sg cache     |
|                  |      |                 |
+--------+---------+      +-------+---------+
         |                        |
         v                        |
+--------+---------+              |
| sg._add_alu      |              |
| (alu=21, hlu=xxx)|              |
|                  |              |
+--------+---------+              |
         |                        v
         |                +-------+---------+
         |                | sg.update()     |
         |                | {21: xxx} is in |
         |                | sg.alu_hlu_map  |
         v                +-------+---------+
+--------+---------+              |
| sg._delete_hlu   |              |
| (alu=21)         |              |
| lun(21).delete() |              v
+------------------+      +-------+---------+
                          | lun created with|
                          | same alu=21     |
                          +-------+---------+
                                  |
                                  |
                                  |
                                  v
                          +-------+---------------+
                          | sg.attach_alu(alu=21) |
                          |                       |
                          | !!!cache dirty!!!     |
                          | Nothing will do on VNX|
                          | because alu=21 is     |
                          | found in the cache    |
                          |                       |
                          +-----------------------+