Request canceled when waiting on container VM process to launch

[jzt]

VIC version:

0.8

Deployment details:

What was the vic-machine create command used to deploy the VCH?

vic-machine-windows create -t https://<target ip/path> -u <user> --thumbprint <thumbprint> -r <compute resource> \
-n <name> --appliance-cpu 2 appliance-memory=8192 -i <image store> --vs <volume store>:default -b <bridge network> --pn <public network> --mn <management network> \
--no-tlsverify --registry-ca ca.crt --insecure-registry <registry ip> -f

Steps to reproduce:
docker -H <VCH IP>:2376 --tls run -i --cpuset-cpus 2 --memory 4g --name rhelbase <registry ip>/library/rhel6.8

Actual behavior:

Unable to find image '<registry ip>/library/rhel6.8:latest' locally
Pulling from library/rhel6.8
1ebabfe4fec3: Pull complete
e530241300b2: Pull complete
Digest: sha256:83eb517e937a195c2efc23485c341629f4c0f0ad9d845e7e705b2b3f98dd4f1d
Status: Downloaded newer image for library/rhel6.8:latest
docker: Error response from daemon: Server error from portlayer: unable to wait for process launch status: Post https://<VCH IP>/sdk: net/http: request canceled.

Expected behavior:
Container deployed without error.

Additional details as necessary:
Reproduction done using docker client version 1.10.3 (the same as the user) but did not end with the user's issue (to check whether or not it is related to the docker client).

User was able to deploy the same container via Admiral. Admiral uses a Java SDK to call the Docker API instead of the docker client itself, so we asked the user to run a last test:

Deploy a new VCH with the API running on port 2375 (without SSL), then capture the network traffic with tcpdump on the admiral machine and deploy a container on the new VCH from the Admiral Web GUI.

The second step would be to capture with tcpdump on the docker client machine and deploy the container from this machine.

[dougm]

"net/http: request canceled" is from the stdlib net/http client and happens in this case when we hit the 3 minute propertyCollectorTimeout set here: https://github.com/vmware/vic/blob/master/lib/portlayer/exec/base.go#L186

[jzt]

I'm currently in the process of trying to repro the issue. We are definitely hitting the 3 minute timeout as @dougm suggested, although the vpxd log window provided in the bundle is well after the window in which the incident occurred, and there are no other indications that I've been able to see in the logs as to why it happened.

[jzt]

Closing in favor of #4613, which contains more useful info about this issue.