VCH Wizard hangs on Network section

[emeirell]

I recently upgrade my VIC environment to version 1.4, including vSphere plugin.
When trying to create a new VCH the wizard hangs on the Network section with an endless spinning.

VIC: 1.4.0.1149
vCenter Appliance: 6.5.0.5300
Browser: Chrome 61.0.3163.100

The environment has 2 cluster, management and workload, both of them are using vSphere Distributed switches with Port Groups and Logical Switches.

Got the followin error fro Chrome Dev Tools:
VM4731 1.bc50ae8091d8d0c74ed0.chunk.js:1 TypeError: Cannot read property 'dvsHosts' of null
at VM4714 0.02e8893b931a1b42c4d5.chunk.js:1
at Array.map ()
at e.project (VM4714 0.02e8893b931a1b42c4d5.chunk.js:1)
at e._next (VM4704 main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.E9/g.e.next (VM4704 main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.checkIterators (VM4704 main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.notifyNext (VM4704 main.1d9fb411ca52796f3c7e.bundle.js:1)
at e._next (VM4704 main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.E9/g.e.next (VM4704 main.1d9fb411ca52796f3c7e.bundle.js:1)
at e._next (VM4704 main.1d9fb411ca52796f3c7e.bundle.js:1)

[Andy-Knight]

Yep, i pulled 1.4.0 from the VMware official download page an hour ago and performed a greenfield install. I have the same issue when deploying a VCH through the wizard.

My environment has one cluster (2 x 6.0 ESXi hosts), no standalone hosts, NSX 6.3.1, 1 x DvSwitch containing 4 NSX logical switches, 1 x standard port group.

VIC: 1.4.0.1109
vCenter Appliance: 6.5.0.5300
Browser: Firefox 60.0.1
Client OS: Mac Sierra 10.12.6

[javierfz1980]

This seems to be a known issue...

@pdaigle has found the same issue on his environment last week. Patrick's words, and same log (from viclifecylce channel):

"Hitting a bug with vic-ui in my lab. It starts up fine but when I get to the “Configure Networks” step, it just hangs (blue spinning wheel)."

"TypeError: Cannot read property 'dvsHosts' of null
at 0.02e8893b931a1b42c4d5.chunk.js:1
at Array.map ()
at e.project (0.02e8893b931a1b42c4d5.chunk.js:1)
at e._next (main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.next (main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.checkIterators (main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.notifyNext (main.1d9fb411ca52796f3c7e.bundle.js:1)
at e._next (main.1d9fb411ca52796f3c7e.bundle.js:1)
at e.next (main.1d9fb411ca52796f3c7e.bundle.js:1)
at e._next (main.1d9fb411ca52796f3c7e.bundle.js:1)
(anonymous) @ 1.bc50ae8091d8d0c74ed0.chunk.js:1"

As a workaround he mentioned that Upgrading to 6.5.0.20000 seems to fix the issue.

We will have to figure out what is wrong anyway… I’ll take a look as soon as I can.

[javierfz1980]

Please provide VC and ESXs build numbers...

Do we have a working environment were I can reproduce the error ?

Thanks.

[Andy-Knight]

I updated my vCenter to 6.5.0.5700 Build 7119070 (6.5.0f) and left my hosts at 6.0.0, build 3620759 (my hosts aren't supported for 6.5). Still the same issue.

[javierfz1980]

Thanks for the info Andy, I’ll take a look with those vc builds and I’ll let you know my findings.

[javierfz1980]

I've been able to reproduce the issue with following environment configuration (vic 1.4.0):

• https://buildweb.eng.vmware.com/ob/7119070/
• vCenter Server 6.5 f (EP3-2)
• version 6.5.0.25940
• build 7119070

However everything is working fine on my local environment configuration (vic 1.4.0):

• https://buildweb.eng.vmware.com/ob/7515524/
• vCenter Server 6.5 U1e (EP5)
• version 6.5.0.63200
• build 7515524

The problem is in this request (introduced on 19/01/2018):

vic-ui/h5c/vic/src/vic-webapp/src/app/create-vch-wizard/create-vch-wizard.service.ts

Line 297 in a4f1012

return this.http.get(`/ui/data/properties/${dv['objRef']}?properties=dvs:dvsHostsData`)

'ui/data/properties/urn:vmomi:VmwareDistributedVirtualSwitch:dvs-22:2bf92a7f-bf5b-46dd-9dd1-d9d800296129?properties=dvs:dvsHostsData'

The above request is returning null on vc 6.5 f (EP3-2) environment:

The same request is returning the expected data on vc 6.5 U1e (EP5) environment:

Our doc states 6.5.0d or later (but I'm not sure if this is correct based on above information)
https://vmware.github.io/vic-product/assets/files/html/1.4/vic_vsphere_admin/vic_installation_prereqs.html

Updating VC to a newer version and reinstalling plugin should fix the problem. Version 6.5 U1e (EP5) will fix the problem for sure, and maybe an earlier one too.

[zjs]

@jak-atx:

How can I map the URIs from #504 (comment) to the corresponding vSphere API call?

Also, do we have any request/response logging for our outgoing API calls? Being able to explain how to reproduce this outside of VIC would be helpful when talking to other teams.

I'd like a crisp example of the behavior that changed between the vSphere versions to track down when that change was made (so we can clearly communicate to users when we will and won't see this issue) and what workarounds might exist.

[jak-atx]

@javierfz1980 can you please list the corresponding network call(s)? I don't believe we are logging these anywhere currently.

[javierfz1980]

@zjs @jak-atx
Guys, sadly we are not logging anything... the specific network call is made here:

vic-ui/h5c/vic/src/vic-webapp/src/app/create-vch-wizard/create-vch-wizard.service.ts

Line 297 in a4f1012

return this.http.get(`/ui/data/properties/${dv['objRef']}?properties=dvs:dvsHostsData`)

And the call is made in the context of the execution of the following service method:

vic-ui/h5c/vic/src/vic-webapp/src/app/create-vch-wizard/create-vch-wizard.service.ts

Line 282 in a4f1012

getDistributedPortGroups(dcObj: ComputeResource, resourceObjName?: string): Observable<any> {

[renmaosheng]

vic-ui/h5c/vic/src/vic-webapp/src/app/create-vch-wizard/create-vch-wizard.service.ts

Line 305 in a4f1012

return dvsHostsData['dvs:dvsHostsData']['dvsHosts'];

Sometimes, the VDS query API returns null value for field of dvsHostsData['dvs:dvsHostsData'],
in that situation, it will throw Type Error if you try to fetch the inner field of dvsHostsData['dvs:dvsHostsData']['dvsHosts'];

we need add null value check logic in line 305:
e.g:
if(dvsHostsData['dvs:dvsHostsData'] !=null) {
dvsHostsData['dvs:dvsHostsData']['dvsHosts'];
} else {
....
}
if some dvs doesn't have ESXi hosts connected to, the value of "dvsHostsData['dvs:dvsHostsData']" will be NULL for some VC version, the value can also be {dvsHosts[]}. That's why we observed differently due to VC version.

[renmaosheng]

[javierfz1980]

@renmaosheng
the null check it's ok to prevent the console to popup the error but it won't fix the real issue, the wizard will still hang on the network step since if portgroups are not retrieved you can't select them and you can't proceed to the next step of the wizard.

The real issue is not the null or {} return, we can handle that... the real issue is that the endopint is returning null or {} when it should return the connwcted hosts list.

[renmaosheng]

if it is null, that means there is no ESXi hosts connected to that switch, we can't even put the vch VM on the portgroup of that switch as there is no compute resources, we should filter out the portgroup in UI, not shown them.

[javierfz1980]

I know tha null or {} means there are no hosts connected to the vds, and that is the problem... because we know that there are host connected to that vds. the response shouldn’t be null or {}. It should be the list of host.

As I said at the beginning ... I’m quering the same environment with the same vds connected to the same hosts from two different UI client versions. One UI client is retrieving the list of connected host as you can see in the screens (so the host are there and they are connected) and the other one connected to the same vc containing the same vds is not retrieving the list of hosts...

NOTE: That said, I agree that we should add a null check to prevent cases were there are nos host connected to the vds. But it won’t fix this, because we know there are host connected to the vds.

[javierfz1980]

I've created a brand new environment were issue is reproducible:
(It has a lease of 7 days, please let me know if you need to extend it.)
https://10.160.193.177/

Here you have some screenshots of the issue:

current vds hosts list:

vch wizard resource selection:

console error:

vds hosts request:

vds hosts request response:

As you can see in the 1st screenshot, host 10.192.218.83 is connected to the vds and the response shouldn’t be null or {}. It should be a list containing host 10.192.218.83.

[zjs]

Thanks, @javierfz1980. To investigate possible changes between the underlying infrastructure versions, it would be helpful to track down what is happening within the /ui/data/properties/ call. That appears to be something we've implemented:

vic-ui/h5c/vic-service/src/main/java/com/vmware/vic/mvc/DataAccessController.java

Lines 77 to 112 in e8e3267

	/**
	* Generic method to fetch properties for a given object.
	* e.g. /properties/{objectId}?properties=name,config
	*
	* @param encodedObjectId
	* Encoded object id.
	*
	* @param properties
	* Properties passed as a request parameter that needs to be fetched.
	* They are comma separated.
	* For e.g. name,runtime
	*
	* @return
	* Returns a map with property name as key and property value as the value.
	*/
	@RequestMapping(value = "/properties/{objectId}")
	@ResponseBody
	public Map<String, Object> getProperties(
	@PathVariable("objectId") String encodedObjectId,
	@RequestParam(value = "properties", required = true) String properties,
	@RequestParam(value = "page", defaultValue = "1") int page,
	@RequestParam(value = "pageSize", defaultValue = "10") int pageSize)
	throws Exception {
	Object ref = getDecodedReference(encodedObjectId);
	String objectId = _objectReferenceService.getUid(ref);
	String[] props = properties.split(",");
	PropertyValue[] pvs = QueryUtil.getProperties(_dataService, ref, props);
	Map<String, Object> propsMap = new HashMap<String, Object>();
	propsMap.put(OBJECT_ID, objectId);
	for (PropertyValue pv : pvs) {
	propsMap.put(pv.propertyName, pv.value);
	}
	return propsMap;
	}

Once I can express the root cause of the issue without referring to our code, I can reach out to the people responsible for the code that seems to be misbehaving and ask for their help tracking down what's happening and whether there's a way we could workaround the issue 🙂

[javierfz1980]

@zjs thanks for your answer !
I'm not an expert on the vic-service, but as AFAIK, the endpoint that you are pointing is not handling the failing vds host request...

@jak-atx please correct me if I'm wrong, but our vic-service pointed by @zjs in the previous comment is handling requests made at /ui/vic/rest/data/properties

vic-ui/h5c/vic-service/src/main/java/com/vmware/vic/mvc/DataAccessController.java

Lines 77 to 112 in e8e3267

	/**
	* Generic method to fetch properties for a given object.
	* e.g. /properties/{objectId}?properties=name,config
	*
	* @param encodedObjectId
	* Encoded object id.
	*
	* @param properties
	* Properties passed as a request parameter that needs to be fetched.
	* They are comma separated.
	* For e.g. name,runtime
	*
	* @return
	* Returns a map with property name as key and property value as the value.
	*/
	@RequestMapping(value = "/properties/{objectId}")
	@ResponseBody
	public Map<String, Object> getProperties(
	@PathVariable("objectId") String encodedObjectId,
	@RequestParam(value = "properties", required = true) String properties,
	@RequestParam(value = "page", defaultValue = "1") int page,
	@RequestParam(value = "pageSize", defaultValue = "10") int pageSize)
	throws Exception {
	Object ref = getDecodedReference(encodedObjectId);
	String objectId = _objectReferenceService.getUid(ref);
	String[] props = properties.split(",");
	PropertyValue[] pvs = QueryUtil.getProperties(_dataService, ref, props);
	Map<String, Object> propsMap = new HashMap<String, Object>();
	propsMap.put(OBJECT_ID, objectId);
	for (PropertyValue pv : pvs) {
	propsMap.put(pv.propertyName, pv.value);
	}
	return propsMap;
	}

The failing vds hosts request (mentioned by me on the previous comments/screenshots) is trying to fetch data from /ui/data/properties and AFAIK this is a vcenter endpoint and is not under our scope (again I'm not an expert on the vic-service, so correct me if I'm wrong).

[zjs]

The failing vds hosts request (mentioned by me on the previous comments/screenshots) is trying to fetch data from /ui/data/properties and AFAIK this is a vcenter endpoint and is not under our scope (again I'm not an expert on the vic-service, so correct me if I'm wrong).

Is this the only place we do this, or is this a common pattern within our UI code? I think we need to update any code that's calling /ui/data/properties to call /ui/vic/rest/data/properties or something else we've implemented in our service layer.

[javierfz1980]

Yes, you are totally right @zjs .
I've been talking with @renmaosheng @jak-atx and Tony Ganchev from Sofia this morning and it looks like we are fetching data from the wrong endpoints. Not sure why, because these calls were there from time before I joined the team, and from that point I've been doing some request to the same endpoints, so we will have a lot of requests to be refactored (17 total by current count + a couple more from the reconfigure feature branch wip)

The plan on this is to start by this issue, in order to fix it ASAP.

Then we will create an epic to make the necessary changes request by request.

I believe that I will have this fixed in the next couple of days. I'll let you know.