Docker 3.1: Volume does not delete if its mounting fails

[sonawane-shashikant]

Test Bed -
OS - CentOS
3PAR version - Release version 3.3.1 (MU4)
Plugin - "PluginReference": "docker.io/hpestorage/hpedockervolumeplugin:3.1"

Steps to reproduce -
1.Create normal docker volume and pass invalid -o fsOwner="1001-1001"
2.Try to mount this volume to see mount failure.
3. Verify the VLUNs entries on 3PAR.
3. Now try to delete the volume.

Expected Result - In case of mount failure, VLUN entries on 3PAR should also be cleared and docker volume should get deleted successfully.

Actual Result - VLUN entries remains on 3PAR even when mounting fails. Volume does not delete saying, "unable to remove volume: remove : resource in use"

[docker@cld6b10 ~]$ docker volume ls
DRIVER VOLUME NAME
[docker@cld6b10 ~]$ docker volume create -d hpe --name=INVALID-FS-OWNER -o fsOwner="Shashi-Shashi" -o fsMode="07-77" -o size=9 -o backend=3PAR Error response from daemon: create INVALID-FS-OWNER: Invalid value '07-77' specified for fsMode. Please specify an integer value.
[docker@cld6b10 ~]$ docker volume create -d hpe --name=INVALID-FS-OWNER -o fsOwner="Shashi-Shashi" -o fsMode="0777" -o size=9 -o backend=3PAR
INVALID-FS-OWNER
[docker@cld6b10 ~]$ docker volume inspect INVALID-FS-OWNER
[
{
"Driver": "hpe:latest",
"Labels": {},
"Mountpoint": "/var/lib/docker/plugins/8564fb4e9a950e34333f8f5c1d8db4cbef0138416dc0143b14ee860fc59094a6/rootfs",
"Name": "INVALID-FS-OWNER",
"Options": {
"backend": "3PAR",
"fsMode": "0777",
"fsOwner": "Shashi-Shashi",
"size": "9"
},
"Scope": "global",
"Status": {
"volume_detail": {
"3par_vol_name": "dcv-WP2hIcOWT9y5K40p1.Un3A",
"backend": "3PAR",
"compression": null,
"cpg": "SSD_r1",
"domain": "TEST_DOMAIN",
"flash_cache": null,
"fsMode": "0777",
"fsOwner": "Shashi-Shashi",
"mountConflictDelay": 30,
"provisioning": "thin",
"size": 9,
"snap_cpg": "SSD_r1"
}
}
}
]
[docker@cld6b10 ~]$ docker run -it -d -v INVALID-FS-OWNER:/data --volume-driver hpe --name MOUNTER --rm busybox /bin/sh
3cc02a91b1a7e53755a8b935a8cdaa7dff8f492a22a03a7cd7035e6a63d9b13b
docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/8564fb4e9a950e34333f8f5c1d8db4cbef0138416dc0143b14ee860fc59094a6/rootfs': Post http://%2Frun%2Fdocker%2Fplugins%2F8564fb4e9a950e34333f8f5c1d8db4cbef0138416dc0143b14ee860fc59094a6%2Fhpe.sock/VolumeDriver.Mount: net/http: request canceled (Client.Timeout exceeded while awaiting headers).
[docker@cld6b10 ~]$

[docker@cld6b10 ~]$ docker volume rm INVALID-FS-OWNER
Error response from daemon: unable to remove volume: remove INVALID-FS-OWNER: resource in use
[docker@cld6b10 ~]$

3PAR output after mount failure -

CSIM-8K02_MXN6072AC7 cli% showvv -cpg SSD_r1 -showcols Name,Comment
Name Comment
dcv-WP2hIcOWT9y5K40p1.Un3A {"volume_id": "58fda121-c396-4fdc-b92b-8d29d7e527dc", "name": "58fda121-c396-4fdc-b92b-8d29d7e527dc", "type": "Docker", "display_name": "INVALID-FS-OWNER"}

total
CSIM-8K02_MXN6072AC7 cli% showvlun -v dcv-WP2hIcOWT9y5K40p1.Un3A
Active VLUNs
Lun VVName HostName --------Host_WWN/iSCSI_Name-------- Port Type Status ID
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 iqn.1994-05.com.redhat:d0f7bb324eaa 2:2:1 matched set active 1
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 iqn.1994-05.com.redhat:d0f7bb324eaa 3:2:1 matched set active 1

2 total

VLUN Templates
Lun VVName HostName -Host_WWN/iSCSI_Name- Port Type
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 ---------------- 2:2:1 matched set
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 ---------------- 3:2:1 matched set

2 total
CSIM-8K02_MXN6072AC7 cli% showvlun -v dcv-WP2hIcOWT9y5K40p1.Un3A
Active VLUNs
Lun VVName HostName --------Host_WWN/iSCSI_Name-------- Port Type Status ID
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 iqn.1994-05.com.redhat:d0f7bb324eaa 2:2:1 matched set active 1
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 iqn.1994-05.com.redhat:d0f7bb324eaa 3:2:1 matched set active 1

2 total

VLUN Templates
Lun VVName HostName -Host_WWN/iSCSI_Name- Port Type
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 ---------------- 2:2:1 matched set
0 dcv-WP2hIcOWT9y5K40p1.Un3A cld6b10 ---------------- 3:2:1 matched set

2 total
CSIM-8K02_MXN6072AC7 cli%

[wdurairaj]

Fix available in hpestorage/legacyvolumeplugin:3.1 and hpestorage/hpedockervolumeplugin:3.1

[sonawane-shashikant]

Since fsOwner validation is implemneted in the latest plugin, this bug is verified by mounting volume without multipathd service and making host available before mounting.

1. By stopping the multipathd service

[root@cld6b9 ~]# docker run -it -v INVALID-FS-OWNER:/data --volume-driver hpe --rm busybox /bin/sh
docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/15822b1f3df1f3008b4ed1174e2ba07c99962c58ea4ed987f26b93da481aa636/rootfs': Unexpected error while running command.
Command: multipathd show status
Exit code: 1
Stdout: ''
Stderr: ''.
[root@cld6b9 ~]#

[root@cld6b9 ~]# docker volume inspect INVALID-FS-OWNER
[
{
"Driver": "hpe:latest",
"Labels": {},
"Mountpoint": "/var/lib/docker/plugins/15822b1f3df1f3008b4ed1174e2ba07c99962c58ea4ed987f26b93da481aa636/rootfs",
"Name": "INVALID-FS-OWNER",
"Options": {
"backend": "3PAR",
"fsMode": "0111",
"fsOwner": "1001:1001",
"size": "9"
},
"Scope": "global",
"Status": {
"volume_detail": {
"3par_vol_name": "dcv-X1B09OmsRfKXhIrx-85szg",
"backend": "3PAR",
"compression": null,
"cpg": "SHASHI-SSD",
"domain": "SUMIT",
"flash_cache": null,
"fsMode": "0111",
"fsOwner": "1001:1001",
"mountConflictDelay": 30,
"provisioning": "thin",
"size": 9,
"snap_cpg": "SHASHI-SSD"
}
}
}
]
[root@cld6b9 ~]#

CSIM-8K02_MXN6072AC7 cli% showvlun -v dcv-X1B09OmsRfKXhIrx-85szg
Active VLUNs
no vluns listed

VLUN Templates
no vluns listed
CSIM-8K02_MXN6072AC7 cli%

---

2. By making host available before mounting

[root@cld6b9 ~]# docker run -it -v INVALID-FS-OWNER:/data --volume-driver hpe --rm busybox /bin/sh
docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/15822b1f3df1f3008b4ed1174e2ba07c99962c58ea4ed987f26b93da481aa636/rootfs': HPE Docker Volume Plugin Mount Failed: ('Initialize Connection Failed: connection info retrieval failed, error is: ', 'Forbidden (HTTP 403) 118 - Volume not in same domain').
[root@cld6b9 ~]#

CSIM-8K02_MXN6072AC7 cli% showvlun -v dcv-X1B09OmsRfKXhIrx-85szg
Active VLUNs
no vluns listed

VLUN Templates
no vluns listed
CSIM-8K02_MXN6072AC7 cli%