Do not see the old volumes after performing storage vMotion of the vm

[ashahi1]

Steps:

1. Create a volume on sharedVmfs-0 datastore. VM is on shared-Vmfs-0 datastore.
2. Run a container with volume attached
3. While the container is running, migrate the vm to a different datastore - storage vMotion
4. Docker volume ls and Admin cli volume ls does not list the volume created before migration.

Logs have been attached at the end.

---

Steps and their output are as follows:

1. Created a volume named testVolXXX

root@sc-rdops-vm02-dhcp-52-237:~# docker volume create -d vsphere:latest --name testVolXXX
testVolXXX
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      testVolXXX@sharedVmfs-0
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker volume inspect testVolXXX@sharedVmfs-0
[
    {
        "Driver": "vsphere:latest",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/testVolXXX@sharedVmfs-0",
        "Name": "testVolXXX@sharedVmfs-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "13MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Fri Jul 21 19:07:53 2017",
            "created by VM": "ubuntu-VM0.0",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        }
    }
]
root@sc-rdops-vm02-dhcp-52-237:~#

2. Admin cli volume ls

[root@sc-rdops-vm04-dhcp-109-225:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume      Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to   Access      Attach-as               Created By    Created Date
----------  ------------  --------  --------  ----  ----------  ------  -----------  ------------  ----------  ----------------------  ------------  ------------------------
testVolXXX  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         ubuntu-VM0.0  read-write  independent_persistent  ubuntu-VM0.0  Fri Jul 21 19:07:53 2017

3. Ran a container with volume attached.

root@sc-rdops-vm02-dhcp-52-237:~#  docker run -it --volume-driver=vsphere -v testVolXXX@sharedVmfs-0:/vol1 --name ub_XXX ubuntu
root@40409f0829e8:/#

4. While container is running, migrated the vm to a sharedVmfs-1.

5. Docker volume ls and admin cli volume ls does not lists any volumes.

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
root@sc-rdops-vm02-dhcp-52-237:~#

[root@sc-rdops-vm04-dhcp-109-225:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume  Datastore  VMGroup  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access  Attach-as  Created By  Created Date
------  ---------  -------  --------  ----  ----------  ------  -----------  -----------  ------  ---------  ----------  ------------

[root@sc-rdops-vm04-dhcp-109-225:~]

Logs:
docker-volume-vsphere.txt
vmdk_ops.txt

[lipingxue]

Checked vmdk_ops.txt.
After migration, we do receive another attach request from the same VM.

07/21/17 19:10:45 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] *** disk_attach: VMDK /vmfs/volumes/sharedVmfs-0/dockvols/_DEFAULT/testVolXXX.vmdk to VM 'ubuntu-VM0.0' , bios uuid = 564d9ddb-5e8a-844b-4247-6a33944412b1, VC uuid=52693e9f-79d8-9b9a-0350-aa212f770b45)
07/21/17 19:10:45 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [WARNING] Failed to find VM by VC UUID 52693e9f-79d8-9b9a-0350-aa212f770b45, trying BIOS UUID 564d9ddb-5e8a-844b-4247-6a33944412b1
07/21/17 19:10:45 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] Attaching /vmfs/volumes/sharedVmfs-0/dockvols/_DEFAULT/testVolXXX.vmdk as independent_persistent
07/21/17 19:10:46 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] Disk /vmfs/volumes/sharedVmfs-0/dockvols/_DEFAULT/testVolXXX.vmdk successfully attached. controller pci_slot_number=160, disk_slot=5
07/21/17 19:10:46 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] executeRequest 'attach' completed with ret={'Unit': '5', 'ControllerPciSlotNumber': '160'}
07/21/17 19:10:53 236486 [MainThread] [INFO   ] Connecting to the local Service Instance as 'dcui' 
07/21/17 19:10:54 236486 [MainThread] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
07/21/17 19:11:00 120912 [MainThread] [INFO   ] Started new thread : 782454302464 with target <function execRequestThread at 0xb62c47a9d8> and args (11, 235223, b'{"cmd":"list","details":{"Name":""},"version":"2"}')
07/21/17 19:11:00 120912 [Thread-111] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
07/21/17 19:11:01 120912 [ubuntu-VM0.0-nolock-list] [INFO   ] executeRequest 'list' completed with ret=[{'Name': 'testVolXXX@sharedVmfs-0', 'Attributes': {}}]   -----> here the list complete successfully.
07/21/17 19:13:15 120912 [VMChangeListener] [INFO   ] VMChangeListener: VM was powered down and then deleted right away. Fault msg: The object 'vim.VirtualMachine:14' has already been deleted or has not been completely created    ---->here seems the VM migrate not finished yet.
07/21/17 19:13:45 237208 [MainThread] [INFO   ] Connecting to the local Service Instance as 'dcui' 
07/21/17 19:13:46 237208 [MainThread] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
07/21/17 19:14:39 237357 [MainThread] [INFO   ] Connecting to the local Service Instance as 'dcui' 
07/21/17 19:14:40 237357 [MainThread] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
07/21/17 19:23:10 120912 [MainThread] [INFO   ] Started new thread : 782454302464 with target <function execRequestThread at 0xb62c47a9d8> and args (13, 236673, b'{"cmd":"list","details":{"Name":""},"version":"2"}')
07/21/17 19:23:10 120912 [Thread-112] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
07/21/17 19:23:11 120912 [ubuntu-VM0.0-nolock-list] [INFO   ] executeRequest 'list' completed with ret=[]   -----> here list cmd return with empty

[lipingxue]

@ashahi1 Could you check in the datastore /vmfs/volumes/sharedVmfs-0, does the vmdk file for volume "testVolXXX" exist or not?

[ashahi1]

@lipingxue No, I do not see the VMDKs at /vmfs/volumes/sharedVmfs-0

[lipingxue]

I checked the Anup's setup (after migration).
In "/vmfs/volumes/sharedVmfs-0/dockervol", the directory to store volumes for _DEFAULT vmgroup is missing.

ls /vmfs/volumes/sharedVmfs-0/dockvols/ -al
total 1040
drwxr-xr-x    1 root     root           420 Jul 21 23:13 .
drwxr-xr-t    1 root     root          2800 Jul 21 23:24 ..
lrwxrwxrwx    1 root     root            72 Jul 21 22:26 _DEFAULT -> /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111

The symbol link "_DEFAULT" is there, but the actual directory /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111 is missing.

We need to see why "/vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111" and vmdk files under it is missing.

[govint]

What's the behavior if the volume is attached and then svmotion is done?

And if they volume is created like volume@datastore it should be accessible always. What happens if volume is created this way, then attached to the VM and then svmotion

[govint]

The volume will be in the vm's folder

[lipingxue]

@govint I did the following test

1. ubuntu-VM0.1 is in sharedVmfs-0, and I create a volume from this VM

root@sc-rdops-vm02-dhcp-52-237:~# docker volume create -d vsphere:latest --name testVolXXX@sharedVmfs-1
testVolXXX@sharedVmfs-1
root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      testVolXXX@sharedVmfs-1

2. start a container to use this volume

root@sc-rdops-vm02-dhcp-52-237:~# docker run -it --volume-driver=vsphere -v testVolXXX@sharedVmfs-1:/vol1 --name ub_XXX ubuntu

3. run svmotion to migrate ubuntu-VM0.1 from sharedVmfs-0 to sharedVmfs-1.
4. After svmotion, the volume is still visible, but it also shows some volumes which stores VM info

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      testVolXXX@sharedVmfs-1
vsphere:latest      ubuntu-VM0.1_1@sharedVmfs-1
vsphere:latest      ubuntu-VM0.1_2@sharedVmfs-1
vsphere:latest      ubuntu-VM0.1_4@sharedVmfs-1

[lipingxue]

@govint Yes, you are right. I tried to repro based on the steps Anup metioned, the volume "testVolXXX" is migrated to datastore "shared-Vmfs-1" along with the vmdk file which represent the VM itself. The volume is renamed and put into the VM directory on "shared-Vmfs-1". So no volume in "/vmfs/volumes/shared-Vmfs-0/dockervols " and no volume in "/vmfs/volumes/shared-Vmfs-1/dockervols. That is why "vmdkops_admin volume ls" and "docker volume ls" shows no volume after svmotion.

@msterin @pdhamdhere I think it is the expected behavior of svmotion, and we should document this in the known issue. Any comments?

[msterin]

we know svmotion messes up the locations and the names of attached vmdks, so yes, this is an expected behavior and yes. we should document it.