Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional edge gateway external network features: multiple external networks, manual IP assignment, IP pool sub-allocation #323

Closed
okozarva opened this issue Sep 5, 2019 · 37 comments · Fixed by #401
Assignees

Comments

@okozarva
Copy link

okozarva commented Sep 5, 2019

Terraform Version

Terraform v0.12.5

  • provider.vcd v2.4.0

Affected Resource(s)

Please list the resources as a list, for example:

  • vcd_edgegateway

Expected Behavior

  • We have multiple subnets in the external network. But, there is no option to which IP and gateway will be use when creating edge gateway.

1
2

  • Secondly, there is no option for sub allocate ip.

3

@Didainius
Copy link
Collaborator

Partly related to https://github.com/terraform-providers/terraform-provider-vcd/issues/308 (could be done together)

@Didainius
Copy link
Collaborator

Hi @okozarva, thanks for the issue. I need to clarify a bit with you so that this is clearer when someone starts to work.

You want to be able to manually specify 1 or more IP addresses in different subnets (and external networks) to be attached to edge gateway?

image

image

As well as use sub-allocated ranges in the external network subnets?

image

@okozarva
Copy link
Author

okozarva commented Sep 6, 2019

Hi @Didainius,

Thanks for your quick response.

That's features will be perfect.

  • be able to manually specify 1 or more IP addresses in different subnets

  • be able to use sub-allocated ranges in the external network subnets

@Didainius Didainius changed the title IP Assignment, default gateway, sub-allocate ip Additional edge gateway external network features: multiple external networks, manual IP assignment, IP pool sub-allocation Sep 6, 2019
@carmine73
Copy link

With automatic IP assignment I had an issue creating in parallel two edges on the same external network: the second creation failed because terraform try to use the same address of the first one.
As a workaround I've used:

resource "vcd_edgegateway" "edge2" {
  ...
  depends_on = ["vcd_edgegateway.edge1"]
}

and edges are created with IP1 and IP2=IP1+1

@Didainius
Copy link
Collaborator

Cross-referencing #308 because it would be very cheap to implement it while working on this PR.

@Didainius
Copy link
Collaborator

With automatic IP assignment I had an issue creating in parallel two edges on the same external network: the second creation failed because terraform try to use the same address of the first one.
As a workaround I've used:

resource "vcd_edgegateway" "edge2" {
  ...
  depends_on = ["vcd_edgegateway.edge1"]
}

and edges are created with IP1 and IP2=IP1+1

@carmineesposito73 ,
Can you share a bit more details?. I tried creating 3 edge gateways (IP assigned ip_address = "192.168.30.${53 + count.index}") at the same time and attach all of them to the same external network. I don't hit the parallel problem. What is the version of vCD you have? (tried on 9.5 and 10)

@Didainius
Copy link
Collaborator

With automatic IP assignment I had an issue creating in parallel two edges on the same external network: the second creation failed because terraform try to use the same address of the first one.
As a workaround I've used:

resource "vcd_edgegateway" "edge2" {
  ...
  depends_on = ["vcd_edgegateway.edge1"]
}

and edges are created with IP1 and IP2=IP1+1

@carmineesposito73 ,
Can you share a bit more details?. I tried creating 3 edge gateways (IP assigned ip_address = "192.168.30.${53 + count.index}") at the same time and attach all of them to the same external network. I don't hit the parallel problem. What is the version of vCD you have? (tried on 9.5 and 10)

Doh. I am building a new structure to support manual IP addresses and misread your note. You are using the "dynamic" IPs, but I still can't replicate the issue. Tried creating 4 edge gateways in parallel with automatic IP assignment as well.

@carmine73
Copy link

carmine73 commented Nov 19, 2019

@Didainius I used vcloud 8.20.
If I remember well, I tried to create in parallel an advanced edge and a not advanced edge

@Didainius
Copy link
Collaborator

@Didainius I used vcloud 8.20.
If I remember well, I tried to create in parallel an advanced edge and a not advanced edge

Thanks - I will try that. A heads-up though - we're dropping support for 8.20 in the upcoming 2.6.0 as we have to start using newer version of API.

@lvirbalas
Copy link
Collaborator

Just to clarify, support was dropped in 2.5.0, but it still worked as the used API version didn't change. While with 2.6.0 vCD 8.20 won't work.

@Didainius
Copy link
Collaborator

@carmineesposito73, @okozarva, I have a draft PR (https://github.com/terraform-providers/terraform-provider-vcd/pull/401) which should satisfy your needs, but feedback is very welcome as this was a tricky one due to troublesome API structure.

I'm going to further test it, but initial tests were positive (structure is still subject to change). You can check documentation from my PR branch (https://github.com/terraform-providers/terraform-provider-vcd/blob/31716df8198f01d29ec3a070c99368b43d4731a1/website/docs/r/edgegateway.html.markdown)

@carmine73
Copy link

@Didainius I confirm the issue when edge_1 is not advanced and edge_2 is advanced.
No issue if both are of the same type (whether advanced or not)

@Didainius
Copy link
Collaborator

@carmineesposito73, thanks! I will try to replicate it this way then

Didainius referenced this issue in vmware/go-vcloud-director Nov 29, 2019
…le subnet support (#267)

* Adds settings/proves the capability of edge gateway being able to attach multiple subnets and sub-allocation pools (terraform-providers/terraform-provider-vcd#323 )
* Adds additional missing settings to edge gateway structure
* Converts some bool fields to pointers to better distinguish between empty value and false.
@Didainius
Copy link
Collaborator

@okozarva , @carmineesposito73 - this PR is merged to master. I'd appreciate if you can give it a spin and report if there are any issues as we are aiming for a a release soon.

@carmine73
Copy link

hi @Didainius, I got this error:

Error: error creating edge gateway: error instantiating a new Edge Gateway: API Error: 400: [ 5dbcaf95-818f-4a1a-8511-f0c18d243f3a ] HTTP 400 Bad Request
 - cvc-complex-type.2.4.d: Invalid content was found starting with element 'FipsModeEnabled'. No child element is expected at this point.

the configuration is the following

resource "vcd_edgegateway" "edge1" {
  org = "${var.vcd_org}"
  vdc = "${var.vcd_vdc}"

  name                        = "${var.vcd_edge_1}"
  description                 = "${var.vcd_edge_1_descr}"
  configuration               = "compact"
  external_network {
    name = "${var.vcd_ext_net_1}"
    subnet {
      ip_address = "${var.ext_net_1_ips[0]}"
      gateway    = "${var.vcd_ext_net_1_gw}"
      netmask    = "${var.vcd_ext_net_1_mask}"
      suballocate_pool {
        start_address = "${var.ext_net_1_ips[1]}"
        end_address   = "${var.ext_net_1_ips[3]}"
      }
    }
  }
  advanced                    = true
  depends_on = ["vcd_org_vdc.vdc1"]
}

I tried with advanced and not advanced gw.
vcloud version is 9.0 with API 29.0.
Here some logs:

2019/12/04 11:09:05 [INFO] API version 29.0.0 satisfies constraints '= 29.0'
. . .
2019/12/04 11:09:09 ######  <EdgeGateway xmlns="http://www.vmware.com/vcloud/v1.5" name="org-test_1_edge_1">
      <Description>edge gw created by terraform</Description>
      <Configuration>
          <GatewayBackingConfig>compact</GatewayBackingConfig>
          <GatewayInterfaces>
              <GatewayInterface>
                  <Name>Ext-Net-Coll-01</Name>
                  <DisplayName>Ext-Net-Coll-01</DisplayName>
                  <Network href="https://10.23.92.173/api/admin/extension/externalnet/d825f218-c401-4b00-ab87-1902f3191ce3" id="urn:vcloud:network:d825f218-c401-4b00-ab87-1902f3191ce3" type="application/vnd.vmware.admin.network+xml" name="Ext-Net-Coll-01"></Network>
                  <InterfaceType>uplink</InterfaceType>
                  <SubnetParticipation>
                      <Gateway>10.23.96.1</Gateway>
                      <Netmask>255.255.255.128</Netmask>
                      <IpAddress>10.23.96.91</IpAddress>
                      <IpRanges>
                          <IpRange>
                              <StartAddress>10.23.96.92</StartAddress>
                              <EndAddress>10.23.96.94</EndAddress>
                          </IpRange>
                      </IpRanges>
                  </SubnetParticipation>
              </GatewayInterface>
          </GatewayInterfaces>
          <EdgeGatewayServiceConfiguration></EdgeGatewayServiceConfiguration>
          <HaEnabled>false</HaEnabled>
          <UseDefaultRouteForDnsRelay>false</UseDefaultRouteForDnsRelay>
          <AdvancedNetworkingEnabled>true</AdvancedNetworkingEnabled>
          <DistributedRoutingEnabled>false</DistributedRoutingEnabled>
          <FipsModeEnabled>false</FipsModeEnabled>
      </Configuration>
  </EdgeGateway>##########################################################################
2019/12/04 11:09:09 Response caller govcd.createEdgeGateway-->govcd.CreateAndConfigureEdgeGatewayAsync-->govcd.executeRequest-->govcd.executeRequest-->govcd.executeRequestCustomErr-->govcd.checkRespWithErrType-->govcd.ParseErr-->govcd.decodeBody
2019/12/04 11:09:09 Response status 400 Bad Request
2019/12/04 11:09:09 ################################################################################
. . .
2019/12/04 11:09:09 Response text: [12000] <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error xmlns="http://www.vmware.com/vcloud/v1.5" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:common="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:ovfenv="http://schemas.dmtf.org/ovf/environment/1" xmlns:vmext="http://www.vmware.com/vcloud/extension/v1.5" xmlns:ns9="http://www.vmware.com/vcloud/networkservice/common/1.0" xmlns:ns10="http://www.vmware.com/vcloud/networkservice/1.0" xmlns:ns11="http://www.vmware.com/vcloud/networkservice/ipam/1.0" xmlns:ns12="http://www.vmware.com/vcloud/versions" stackTrace="javax.ws.rs.BadRequestException: HTTP 400 Bad Request&#xA;&#x9;at org.apache.cxf.jaxrs.utils.SpecExceptions.toBadRequestException(SpecExceptions.java:84)&#xA;&#x9;at org.apache.cxf.jaxrs.utils.ExceptionUtils.toBadRequestException(ExceptionUtils.java:121)&#xA;&#x9;at org.apache.cxf.jaxrs.provider.AbstractJAXBProvider.handleExceptionEnd(AbstractJAXBProvider.java:705)&#xA;&#x9;at org.apache.cxf.jaxrs.provider.AbstractJAXBProvider.handleJAXBException(AbstractJAXBProvider.java:732)&#xA;&#x9;at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:208)&#xA;&#x9;at org.apache.cxf.jaxrs.provider.JAXBElementTypedProvider.readFrom(JAXBElementTypedProvider.java:41)&#xA;&#x9;at org.apache.cxf.jaxrs.provider.JAXBElementTypedProvider.readFrom(JAXBElementTypedProvider.java:34)&#xA;&#x9;at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1348)&#xA;&#x9;at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:1299)&#xA;&#x9;at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:825)&#xA;&#x9;at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:789)&#xA;&#x9;at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:258)&#xA;&#x9;at com.vmware.vcloud.api.rest.jaxrs.ContextClassLoaderInvoker.invoke(ContextClassLoaderInvoker.java:27)&#xA;&#x9;at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)&#xA;&#x9;at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)&#xA;&#x9;at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)&#xA;&#x9;at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)&#xA;&#x9;at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)&#xA;&#x9;at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:262)&#xA;&#x9;at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)&#xA;&#x9;at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)&#xA;&#x9;at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)&#xA;&#x9;at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)&#xA;&#x9;at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)&#xA;&#x9;at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218)&#xA;&#x9;at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)&#xA;&#x9;at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)&#xA;&#x9;at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)&#xA;&#x9;at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)&#xA;&#x9;at com.vmware.vcloud.api.rest.diagnostics.DiagnosticFilter.doFilter(DiagnosticFilter.java:33)&#xA;&#x9;at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:128)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at com.vmware.vcloud.api.framework.web.ExtensibilityFilter.doFilter(ExtensibilityFilter.java:132)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at com.vmware.vcloud.security.filters.ValidityExceptionFilter.doFilterHttp(ValidityExceptionFilter.java:47)&#xA;&#x9;at com.vmware.vcloud.security.filters.HttpFilterBean.doFilter(HttpFilterBean.java:35)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at com.vmware.vcloud.security.web.RestSessionFilter.doFilter(RestSessionFilter.java:209)&#xA;&#x9;at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:128)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at com.vmware.vcloud.security.web.CellLivenessCheckFilter.doFilter(CellLivenessCheckFilter.java:54)&#xA;&#x9;at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:128)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at com.vmware.vcloud.api.rest.versioning.AcceptHeaderFilter.doFilter(AcceptHeaderFilter.java:109)&#xA;&#x9;at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:128)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:39)&#xA;&#x9;at com.vmware.vcloud.security.filters.ConversationFilter$1.run(ConversationFilter.java:37)&#xA;&#x9;at com.vmware.vcloud.common.persist.ConversationContextExecutor.execute(ConversationContextExecutor.java:67)&#xA;&#x9;at com.vmware.vcloud.security.filters.ConversationFilter.doFilter(ConversationFilter.java:45)&#xA;&#x9;at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:128)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at com.vmware.ssdc.backendbase.security.ThreadLocalCleanerFilter.doFilter(ThreadLocalCleanerFilter.java:65)&#xA;&#x9;at com.vmware.vcloud.api.rest.diagnostics.RestApiDiagnosticsInterceptor.doFilter(RestApiDiagnosticsInterceptor.java:128)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)&#xA;&#x9;at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)&#xA;&#x9;at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)&#xA;&#x9;at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)&#xA;&#x9;at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)&#xA;&#x9;at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)&#xA;&#x9;at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301)&#xA;&#x9;at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)&#xA;&#x9;at com.vmware.vcloud.common.web.deployment.impl.RequestDiagnosticFilter.doFilter(RequestDiagnosticFilter.java:96)&#xA;&#x9;at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)&#xA;&#x9;at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)&#xA;&#x9;at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)&#xA;&#x9;at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)&#xA;&#x9;at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)&#xA;&#x9;at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)&#xA;&#x9;at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)&#xA;&#x9;at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)&#xA;&#x9;at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)&#xA;&#x9;at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)&#xA;&#x9;at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)&#xA;&#x9;at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)&#xA;&#x9;at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)&#xA;&#x9;at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)&#xA;&#x9;at org.eclipse.jetty.server.Server.handle(Server.java:499)&#xA;&#x9;at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)&#xA;&#x9;at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)&#xA;&#x9;at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)&#xA;&#x9;at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)&#xA;&#x9;at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)&#xA;&#x9;at java.lang.Thread.run(Thread.java:748)&#xA;Caused by: org.xml.sax.SAXParseException; lineNumber: 30; columnNumber: 11; cvc-complex-type.2.4.d: Invalid content was found starting with element 'FipsModeEnabled'. No child element is expected at this point.&#xA;&#x9;at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)&#xA;&#x9;at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:134)&#xA;&#x9;at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:396)&#xA;&#x9;at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)&#xA;&#x9;at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:284)&#xA;&#x9;at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(XMLSchemaValidator.java:453)&#xA;&#x9;at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.reportSchemaError(XMLSchemaValidator.java:3231)&#xA;&#x9;at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:1795)&#xA;&#x9;at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(XMLSchemaValidator.java:741)&#xA;&#x9;at com.sun.org.apache.xerces.internal.jaxp.validation.ValidatorHandlerImpl.startElement(ValidatorHandlerImpl.java:568)&#xA;&#x9;at com.sun.xml.bind.v2.runtime.unmarshaller.ValidatingUnmarshaller.startElement(ValidatingUnmarshaller.java:101)&#xA;&#x9;at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:246)&#xA;&#x9;at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:180)&#xA;&#x9;at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:415)&#xA;&#x9;at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:394)&#xA;&#x9;at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:190)&#xA;&#x9;... 81 more&#xA;" majorErrorCode="400" message="[ 5dbcaf95-818f-4a1a-8511-f0c18d243f3a ] HTTP 400 Bad Request&#xA; - cvc-complex-type.2.4.d: Invalid content was found starting with element 'FipsModeEnabled'. No child element is expected at this point." minorErrorCode="BAD_REQUEST"/>

I'll try to do some changes to make things working

@Didainius
Copy link
Collaborator

That is great! Thank you for testing. I did see it some time ago, but I thought I have fixed as I did cover my cases.

@carmine73
Copy link

FipsModeEnabled is present since API 30.0

@Didainius
Copy link
Collaborator

I'll check that on vcd 9.0 - this may be the secret about it. I might have not tested it with 9.0 as we haven't reached pre-release testing phase. Although I am using the same API version 29 - it did work on 9.5. Will do some more testing later today and report.

@carmine73
Copy link

carmine73 commented Dec 4, 2019

it works with this "quick and dirty" patch

$ diff -u resource_vcd_edgegateway.go.orig resource_vcd_edgegateway.go
--- resource_vcd_edgegateway.go.orig	2019-12-04 11:25:22.166424741 +0100
+++ resource_vcd_edgegateway.go	2019-12-04 11:31:50.888703791 +0100
@@ -337,7 +337,6 @@
 		Description: d.Get("description").(string),
 		Configuration: &types.GatewayConfiguration{
 			UseDefaultRouteForDNSRelay: takeBoolPointer(d.Get("use_default_route_for_dns_relay").(bool)),
-			FipsModeEnabled:            takeBoolPointer(d.Get("fips_mode_enabled").(bool)),
 			HaEnabled:                  takeBoolPointer(d.Get("ha_enabled").(bool)),
 			GatewayBackingConfig:       d.Get("configuration").(string),
 			AdvancedNetworkingEnabled:  takeBoolPointer(d.Get("advanced").(bool)),
@@ -935,7 +934,6 @@
 	}
 
 	_ = d.Set("use_default_route_for_dns_relay", egw.EdgeGateway.Configuration.UseDefaultRouteForDNSRelay)
-	_ = d.Set("fips_mode_enabled", egw.EdgeGateway.Configuration.FipsModeEnabled)
 	_ = d.Set("advanced", egw.EdgeGateway.Configuration.AdvancedNetworkingEnabled)
 	_ = d.Set("ha_enabled", egw.EdgeGateway.Configuration.HaEnabled)

I'm sorry I use the oldest vCloud version supported :-S

@Didainius
Copy link
Collaborator

It is good. We still support it so you saved me some time on catching the bug early before we have a pre-release testing. I will look at how to make this work.

@carmine73
Copy link

@Didainius I confirm the issue when edge_1 is not advanced and edge_2 is advanced.
No issue if both are of the same type (whether advanced or not)

The issue is no more present, I created edge_1 not advanced and edge_2 advanced in parallel with auto-assigned ip, they got IP and IP+1

@carmine73
Copy link

@Didainius, changing suballocated ip addresses (adding a range or extend the range) requires the edge to be destroyed. Is it as expected?

Terraform will perform the following actions:

  # vcd_edgegateway.edge1 must be replaced
-/+ resource "vcd_edgegateway" "edge1" {
        advanced                        = false
        configuration                   = "compact"
      ~ default_external_network_ip     = "10.23.96.91" -> (known after apply)
      ~ default_gateway_network         = "Ext-Net-Coll-01" -> (known after apply)
        description                     = "edge gw created by terraform"
        distributed_routing             = false
      ~ external_network_ips            = [
          - "10.23.96.91",
        ] -> (known after apply)
        fips_mode_enabled               = false
        fw_default_rule_action          = "deny"
        fw_default_rule_logging_enabled = false
        fw_enabled                      = true
        ha_enabled                      = false
      ~ id                              = "urn:vcloud:gateway:51757609-4401-4ba0-a0bd-8b0c08dc6afe" -> (known after apply)
        lb_acceleration_enabled         = false
        lb_enabled                      = false
        lb_logging_enabled              = false
        lb_loglevel                     = "info"
        name                            = "org-test_1_edge_1"
        org                             = "org-test_1"
      ~ use_default_route_for_dns_relay = false -> (known after apply)
        vdc                             = "org-test_1_PAYG_1"

      - external_network { # forces replacement
          - enable_rate_limit   = false -> null
          - incoming_rate_limit = 0 -> null
          - name                = "Ext-Net-Coll-01" -> null
          - outgoing_rate_limit = 0 -> null

          - subnet {
              - gateway               = "10.23.96.1" -> null
              - ip_address            = "10.23.96.91" -> null
              - netmask               = "255.255.255.128" -> null
              - use_for_default_route = true -> null

              - suballocate_pool {
                  - end_address   = "10.23.96.94" -> null
                  - start_address = "10.23.96.91" -> null
                }
            }
        }
      + external_network { # forces replacement
          + enable_rate_limit   = false
          + incoming_rate_limit = 0
          + name                = "Ext-Net-Coll-01"
          + outgoing_rate_limit = 0

          + subnet {
              + gateway               = "10.23.96.1"
              + ip_address            = "10.23.96.91"
              + netmask               = "255.255.255.128"
              + use_for_default_route = true

              + suballocate_pool {
                  + end_address   = "10.23.96.98"
                  + start_address = "10.23.96.91"
                }
            }
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

@Didainius
Copy link
Collaborator

@carmineesposito73, so far yes. We have yet another issue to fix all edge gateway related updates ( partly mentioned in https://github.com/terraform-providers/terraform-provider-vcd/issues/369). I had to split the PR as it was hard to review because of the structure it implies.

@carmine73
Copy link

I think that can be useful to have automatic suballocation as an option (as it is for edge ip assignment), something like that:

    suballocate_pool {
        ip_addresses  = 4
        use_edge_ip_address = true
    }

use_edge_ip_address = true means that the ip address assigned to the edge interface must be included in the suballocated pool

@Didainius
Copy link
Collaborator

Could you elaborate a bit more what your full workflow would be?

@carmine73
Copy link

carmine73 commented Dec 5, 2019

This is the use case I'm thinking about:
a provider have to create an org vdc with one edge, the organization needs N (random) public ip addresses on the edge for services, there's no need to choose the ip addresses, so automatic allocation can be used.
These are some configuration options:

    # as it works now
    suballocate_pool {
        start_address = "10.23.96.91"
        end_address   = "10.23.96.98"
    }

    # 3 ip addresses taken by unused ip addresses of external network + ip address of edge interface
    # the process to get the ip addresses needed should be the same used to get the edge automatic ip address
    suballocate_pool {
        ip_addresses  = 4
        use_edge_ip_address = true
    }

    # 4 ip addresses taken by unused ip addresses of external network (ip address of edge interface not used for suballocation)
    suballocate_pool {
        ip_addresses  = 4
        use_edge_ip_address = false
    }

    # 'use_edge_ip_address' can be omitted (a default value can be true/false)
    suballocate_pool {
        ip_addresses  = 4
    }
    
    # 8 ip static addresses used ('start_address' and 'end_address' have priority on 'ip_addresses' field)
    # even an ERROR can be ok ('ip_addresses' and 'start_address/end_address' conflict)
    suballocate_pool {
        ip_addresses  = 4           # ignored because there is start_address
        use_edge_ip_address = true  # ignored because there is start_address
        start_address = "10.23.96.91"
        end_address   = "10.23.96.98"
    }

    # ERROR 'end_address' is required if 'start_address' is present
    suballocate_pool {
        ip_addresses  = 4           # ignored becouse there is start_address
        use_edge_ip_address = true  # ignored becouse there is start_address
        start_address = "10.23.96.91"
    }

I don't know if I've answered you question

@Didainius
Copy link
Collaborator

I don't know if I've answered you question

Yes, thanks! Let me think about it and come back.

@Didainius
Copy link
Collaborator

@carmineesposito73 , regarding FIPS mode. Could you try to build it from this branch - https://github.com/Didainius/terraform-provider-vcd/tree/fips-mode-fix ? It does not send the field data when fips_mode_enabled is not set at all. Also should throw a WARNING if you actually set it. It includes all the latest stuff from master repo.

@carmine73
Copy link

great!
tested with advanced and not advanced edge gws

vcd_edgegateway.edge1: Creation complete after 55s [id=urn:vcloud:gateway:cc1f35c2-90a2-48d1-9359-62794faf44ad]

# NOT ADVANCED EDGE GW
$ cat go-vcloud-director.log | grep -A 100 POST | grep -A 35 edgeGateways
2019/12/05 13:06:23 POST https://10.23.92.173/api/admin/vdc/5e351f65-e7cd-42de-b060-e726290f8026/edgeGateways
2019/12/05 13:06:23 --------------------------------------------------------------------------------
2019/12/05 13:06:23 Request data: [1856] <?xml version="1.0" encoding="UTF-8"?>
  <EdgeGateway xmlns="http://www.vmware.com/vcloud/v1.5" name="org-test_1_edge_1">
      <Description>edge gw created by terraform</Description>
      <Configuration>
          <GatewayBackingConfig>compact</GatewayBackingConfig>
          <GatewayInterfaces>
              <GatewayInterface>
                  <Name>Ext-Net-Coll-01</Name>
                  <DisplayName>Ext-Net-Coll-01</DisplayName>
                  <Network href="https://10.23.92.173/api/admin/extension/externalnet/d825f218-c401-4b00-ab87-1902f3191ce3" id="urn:vcloud:network:d825f218-c401-4b00-ab87-1902f3191ce3" type="application/vnd.vmware.admin.network+xml" name="Ext-Net-Coll-01"></Network>
                  <InterfaceType>uplink</InterfaceType>
                  <SubnetParticipation>
                      <Gateway>10.23.96.1</Gateway>
                      <Netmask>255.255.255.128</Netmask>
                      <IpAddress>10.23.96.91</IpAddress>
                      <IpRanges>
                          <IpRange>
                              <StartAddress>10.23.96.91</StartAddress>
                              <EndAddress>10.23.96.94</EndAddress>
                          </IpRange>
                      </IpRanges>
                      <UseForDefaultRoute>true</UseForDefaultRoute>
                  </SubnetParticipation>
                  <UseForDefaultRoute>true</UseForDefaultRoute>
              </GatewayInterface>
          </GatewayInterfaces>
          <EdgeGatewayServiceConfiguration></EdgeGatewayServiceConfiguration>
          <HaEnabled>false</HaEnabled>
          <UseDefaultRouteForDnsRelay>false</UseDefaultRouteForDnsRelay>
          <AdvancedNetworkingEnabled>false</AdvancedNetworkingEnabled>
          <DistributedRoutingEnabled>false</DistributedRoutingEnabled>
      </Configuration>
  </EdgeGateway>

# ADVANCED EDGE GW
$ cat go-vcloud-director.log | grep -A 100 POST | grep -A 35 edgeGateways
2019/12/05 13:14:30 POST https://10.23.92.173/api/admin/vdc/5e351f65-e7cd-42de-b060-e726290f8026/edgeGateways
2019/12/05 13:14:30 --------------------------------------------------------------------------------
2019/12/05 13:14:30 Request data: [1520] <?xml version="1.0" encoding="UTF-8"?>
  <EdgeGateway xmlns="http://www.vmware.com/vcloud/v1.5" name="org-test_1_edge_2">
      <Description>edge gw created by terraform</Description>
      <Configuration>
          <GatewayBackingConfig>compact</GatewayBackingConfig>
          <GatewayInterfaces>
              <GatewayInterface>
                  <Name>Ext-Net-Coll-01</Name>
                  <DisplayName>Ext-Net-Coll-01</DisplayName>
                  <Network href="https://10.23.92.173/api/admin/extension/externalnet/d825f218-c401-4b00-ab87-1902f3191ce3" id="urn:vcloud:network:d825f218-c401-4b00-ab87-1902f3191ce3" type="application/vnd.vmware.admin.network+xml" name="Ext-Net-Coll-01"></Network>
                  <InterfaceType>uplink</InterfaceType>
                  <SubnetParticipation>
                      <Gateway>10.23.96.1</Gateway>
                      <Netmask>255.255.255.128</Netmask>
                      <UseForDefaultRoute>true</UseForDefaultRoute>
                  </SubnetParticipation>
                  <UseForDefaultRoute>true</UseForDefaultRoute>
              </GatewayInterface>
          </GatewayInterfaces>
          <EdgeGatewayServiceConfiguration></EdgeGatewayServiceConfiguration>
          <HaEnabled>false</HaEnabled>
          <UseDefaultRouteForDnsRelay>false</UseDefaultRouteForDnsRelay>
          <AdvancedNetworkingEnabled>true</AdvancedNetworkingEnabled>
          <DistributedRoutingEnabled>false</DistributedRoutingEnabled>
      </Configuration>
  </EdgeGateway>

@carmine73
Copy link

if fips_mode_enabled is used

WARNING! FIPS mode is only supported starting with vCD 9.1. Please do not set this field when using with vCD 9.0

Error: error creating edge gateway: error instantiating a new Edge Gateway: API Error: 400: [ 8439fbf3-af8a-4a8c-8754-a5333871af5e ] HTTP 400 Bad Request
 - cvc-complex-type.2.4.d: Invalid content was found starting with element 'FipsModeEnabled'. No child element is expected at this point.

@Didainius
Copy link
Collaborator

Didainius commented Dec 5, 2019

Thanks! That was my original intention - I did send the request after this warning although I may decide to just bail out when it is run with vCD 9.0 and let the user remove this setting.

@carmine73
Copy link

carmine73 commented Dec 5, 2019

I've realized that automatic ip allocation for edge interface is not a terraform-provider-vdc implementation, but it's just a vcloud feature, since <IpAddress>IP</IpAddress> can be omitted when creating edge

  <SubnetParticipation>
	  <Gateway>10.23.96.1</Gateway>
	  <Netmask>255.255.255.128</Netmask>
	  <UseForDefaultRoute>true</UseForDefaultRoute>
  </SubnetParticipation>

I guess that my request regarding suballocated ip is out-of-the-scope :-(

@Didainius
Copy link
Collaborator

Didainius commented Dec 6, 2019

I've realized that automatic ip allocation for edge interface is not a terraform-provider-vdc implementation, but it's just a vcloud feature, since <IpAddress>IP</IpAddress> can be omitted when creating edge

  <SubnetParticipation>
	  <Gateway>10.23.96.1</Gateway>
	  <Netmask>255.255.255.128</Netmask>
	  <UseForDefaultRoute>true</UseForDefaultRoute>
  </SubnetParticipation|

I guess that my request regarding suballocated ip is out-of-the-scope :-(

I had my thoughts about it and yes - we are not adding any "extra logic" on top of what exists in vCD so implement your above example would require some computations on provider side and that usually bites us back sooner or later.

@carmine73
Copy link

@Didainius, I have to agree with you.
I'll ask vmware to add the feature on vCloud API ;-)

@okozarva
Copy link
Author

It works thank you @Didainius

@carmine73
Copy link

carmine73 commented Dec 17, 2019

@Didainius, just a question, does datasource_vcd_edgegateway.go manage multiple external networks?

I see only one external network block, but the edge has 2 external networks

my-edge1 = {
  "advanced" = false
  "configuration" = "compact"
  "default_external_network_ip" = "10.23.96.91"
  "default_gateway_network" = "Ext-Net-Coll-01"
  "description" = "edge gw created by terraform"
  "distributed_routing" = false
  "external_network" = [
    {
      "enable_rate_limit" = false
      "incoming_rate_limit" = 0
      "name" = "Backup-index-1"
      "outgoing_rate_limit" = 0
      "subnet" = [
        {
          "gateway" = "192.168.55.1"
          "ip_address" = ""
          "netmask" = "255.255.255.0"
          "suballocate_pool" = [
            {
              "end_address" = "192.168.55.3"
              "start_address" = "192.168.55.3"
            },
          ]
          "use_for_default_route" = false
        },
      ]
    },
  ]
  "external_network_ips" = [
    "10.23.96.91",
    "192.168.55.3",
  ]
  "external_networks" = [
    "Ext-Net-Coll-01",
    "Backup-index-1",
  ]
  "ha_enabled" = false
  "id" = "urn:vcloud:gateway:06461a1e-939c-46b9-848c-536a78e9d7fd"
  "name" = "org-template_1_edge_1"
  "org" = "org-template_1"
  "use_default_route_for_dns_relay" = false
  "vdc" = "org-template_1_PAYG_1"
}

here is the extract of api get

        <GatewayInterfaces>
            <GatewayInterface>
                <Name>Ext-Net-Coll-01</Name>
                <DisplayName>Ext-Net-Coll-01</DisplayName>
                <Network href="https://10.23.92.173/api/admin/network/d825f218-c401-4b00-ab87-1902f3191ce3" name="Ext-Net-Coll-01" type="application/vnd.vmware.admin.network+xml"/>
                <InterfaceType>uplink</InterfaceType>
                <SubnetParticipation>
                    <Gateway>10.23.96.1</Gateway>
                    <Netmask>255.255.255.128</Netmask>
                    <IpAddress>10.23.96.91</IpAddress>
                    <IpRanges>
                        <IpRange>
                            <StartAddress>10.23.96.91</StartAddress>
                            <EndAddress>10.23.96.94</EndAddress>
                        </IpRange>
                    </IpRanges>
                    <UseForDefaultRoute>true</UseForDefaultRoute>
                </SubnetParticipation>
                <ApplyRateLimit>false</ApplyRateLimit>
                <UseForDefaultRoute>true</UseForDefaultRoute>
            </GatewayInterface>
            <GatewayInterface>
                <Name>Backup-index-1</Name>
                <DisplayName>Backup-index-1</DisplayName>
                <Network href="https://10.23.92.173/api/admin/network/cd2d3661-ba74-407e-b2f8-1aaf5bcab583" name="Backup-index-1" type="application/vnd.vmware.admin.network+xml"/>
                <InterfaceType>uplink</InterfaceType>
                <SubnetParticipation>
                    <Gateway>192.168.55.1</Gateway>
                    <Netmask>255.255.255.0</Netmask>
                    <IpAddress>192.168.55.3</IpAddress>
                    <IpRanges>
                        <IpRange>
                            <StartAddress>192.168.55.3</StartAddress>
                            <EndAddress>192.168.55.3</EndAddress>
                        </IpRange>
                    </IpRanges>
                    <UseForDefaultRoute>false</UseForDefaultRoute>
                </SubnetParticipation>
                <ApplyRateLimit>false</ApplyRateLimit>
                <UseForDefaultRoute>false</UseForDefaultRoute>
            </GatewayInterface>

@Didainius
Copy link
Collaborator

@Didainius, just a question, does datasource_vcd_edgegateway.go manage multiple external networks?

I see only one external network block, but the edge has 2 external networks

@carmineesposito73, there is a problem in Terraform sdk (from Hashicorp) with multiple typeset computed blocks and sdk fix didn’t get merged for our 2.6 release timeframe (still pending PR hashicorp/terraform-plugin-sdk#197). As soon as terraform sdk is fixed we are going to bump sdk dependency version in our provider and it should start showing multiple external network blocks in future versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants