NIC not connected when cloning VM from template

[ronmessana-concerto]

Terraform Version

Terraform v0.11.3

vSphere Provider Version

provider.vsphere v1.3.0

Affected Resource(s)

• vsphere_virtual_machine

Terraform Configuration Files

provider "vsphere" {
  user           = "${var.vsphere_user}"
  password       = "${var.vsphere_password}"
  vsphere_server = "${var.vsphere_server}"

  # if you have a self-signed cert
  allow_unverified_ssl = true
}

data "vsphere_datacenter" "dc" {
  name = "Datacenter"
}

data "vsphere_datastore" "datastore" {
  name          = "STAGING"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_resource_pool" "pool" {
  name          = "CLUS01/Resources"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_network" "network" {
  name          = "DVOPS"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_virtual_machine" "template" {
  name          = "20180130 - Server 2012R2 Datacenter"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

resource "vsphere_virtual_machine" "vm" {
  name             = "terraform-test"
  resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"
  folder = "DVOPS1690"

  num_cpus = 1
  memory   = 2048
  guest_id = "${data.vsphere_virtual_machine.template.guest_id}"

  scsi_type = "${data.vsphere_virtual_machine.template.scsi_type}"

  network_interface {
    network_id   = "${data.vsphere_network.network.id}"
  }

  disk {
    label            = "disk0"
    size             = "${data.vsphere_virtual_machine.template.disks.0.size}"
    eagerly_scrub    = "${data.vsphere_virtual_machine.template.disks.0.eagerly_scrub}"
    thin_provisioned = "${data.vsphere_virtual_machine.template.disks.0.thin_provisioned}"
  }
  disk {
    label            = "disk1"
    size             = "${data.vsphere_virtual_machine.template.disks.1.size}"
    eagerly_scrub    = "${data.vsphere_virtual_machine.template.disks.1.eagerly_scrub}"
    thin_provisioned = "${data.vsphere_virtual_machine.template.disks.1.thin_provisioned}"
    unit_number = 1
  }
  clone {
    template_uuid = "${data.vsphere_virtual_machine.template.id}"
      
    customize {
        windows_options {
        computer_name  = "terraform-test"
      }
      network_interface {
        ipv4_address = "172.20.90.125"
        ipv4_netmask = 27
        dns_server_list = ["${var.virtual_machine_dns_servers}"]
        
      }

      ipv4_gateway = "172.20.90.97"
    }
  }
}

variable "vsphere_server" {
    description = "vsphere server for the environment - EXAMPLE: vcenter01.hosted.local"
    default = "vcenter01.hosted.local"
}

variable "vsphere_user" {
    description = "vsphere server for the environment - EXAMPLE: vsphereuser"
    default = "vsphereuser"
}

variable "vsphere_password" {
    description = "vsphere server password for the environment"
    default = "vspherep@ssw0rd"
}

variable "virtual_machine_dns_servers" {
  type    = "list"
  default = ["8.8.8.8", "4.2.2.2"]
}

Debug Output

Debug Output:
https://gist.github.com/ronmessana-concerto/c91829184833ccad7991e3665f6cda95

Panic Output

n/a

Expected Behavior

What should have happened? When cloning a VM from a template, the network adapter has the "connected" and "connect at power on" boxes checked.

Actual Behavior

What actually happened? When cloning a VM from a template, the network adapter does not have the "connected" or "connect at power on" boxes checked, leading to failures in configurtion

Steps to Reproduce

1. run 'terraform apply' using the above configuration.

Important Factoids

-VMWare vCenter server 6.0.0 Build 5218300
-When creating a VM from scratch, not using a template, the NIC is always connected.

• Issue exists on both Windows and Linux terraform builds

References

[jason-azze]

@ronmessana-concerto I have a silly question but want to ask to be sure. :-)

Are the "connected" and "connect at power on" checkboxes checked in the template? That is, if you make a clone from the template "manually" using the vSphere client, will the resulting VM have the checkboxes checked and behave as expected?

[ronmessana-concerto]

on the template, "connected" is grayed out and unchecked, since it's powered off, but "connect at power on" is checked. i just ran a "manual" clone from the template, and the "connected" and "connect at power on" are checked as expected. We clone VMs from templates all the time using powershell and manually in the vsphere client without issue.

[vancluever]

@ronmessana-concerto one other question. Is the network that the source template is set to visible to the hosts that you are cloning to?

If it's not, can you try changing that and trying again?

PS: I'm marking this as a bug anyway as this is something that we should probably be fixing anyway during post-clone.

Thanks!

[ronmessana-concerto]

@vancluever The network on the source template is available on all hosts. In the environment, all networks are available on all hosts in the datacenter as a standard.

[sdemura]

I've noticed that if customize is used, then the template will boot with the NIC connected.
If customize is NOT used, then the VM boots as desired, but wait_for_guest_net_timeout =0 must be used in order for terraform apply to complete.

wait_for_guest_net_timeout=0 is not an option for me because I run into #231

[ronmessana-concerto]

@sdemura I'm using the following customize block

customize {
        windows_options {
        computer_name  = "terraform-test"
      }

Do I need to add something additional?

[sdemura]

@ronmessana-concerto I'm using linux, but I add an empty network_interface which is DHCP. I did just notice now that you are trying to use static, though.

customize {
    linux_options { ... }
    network_interface = {}
}

[ronmessana-concerto]

@sdemura I changed my configuration to use DHCP and the results are the same. I don't have any linux templates to test, I'll see about making one and report back

[ronmessana-concerto]

I created a linux template and the NIC is always connected after a clone. The issue appears to be isolated to Windows templates.

[hauntshadow]

Has anyone been able to successfully connect the NIC with a Windows template through Terraform? I'm using v0.11.4 and am getting the network setup successfully, but not connected, which causes a config failure.

[ronmessana-concerto]

I am still experiencing the issue.

[jason-azze]

@hauntshadow Are you also using vCenter 6.0.0?
I don't have the problem with 6.5, and I wonder if that's a factor.
It also might be worth checking to see if the provider is changing your NIC type from, let's say, e1000 in your template to its default choice of VMXNET 3.

[hauntshadow]

I'm using vCenter 6.5. I'll play around with the NIC type. Thanks for the quick replies!

EDIT: Changing the NIC type from e1000 to VMXNET3 worked for me! Thanks @jason-azze for the suggestion!

[jtdroste]

I'm on vCenter 6.0, running into the same issue. I've played around a bit, and found out when you have a customize block, the NIC 'enabled' box gets unchecked after the customize task completes. I actually managed to catch terraform before it applied the customizing VM step, and can confirm that 'enable' was checked, and post customization step, 'enable' was unchecked.

EDIT: Apparently this is a 'feature' of VMware customization - see here. The NIC gets disabled to avoid pulling IP address. You just need to plan around it, apparently...

[emdantrim]

For me, on vCenter 6.5, switching to VMXNET3 (or specifically declaring an interface as such) does not set the device to be connected at power on.

[voki]

For me this issue is totally random, I am cloning CentOS 7.4 template on vCenter 6.0. In most cases network is connected. But on average, every second build of multiple VMs (count=10) ends up with 8 having network connected and 2 not.

[ianmcmahon]

I have this issue consistently. I've created a vm template using packer, building an image of openfiler (based on rPath, which seems loosely centos-ish), and when terraform spins up a vm from this image, it comes up unconnected every time. I haven't been able to find any way to work around this yet.

[vancluever]

@emdantrim sent along some helpful information on this (thanks Emma!) regarding that this is happening with the Ubuntu OVA when customization is set. The OVA is set for DHCP which will cause timeouts, during which period this all may be observable. I'm just noting it here for testing purposes, and it more than likely confirms that this is an intended side-effect of the customization process, as we use Ubuntu OVAs for testing vApp ISO support (sans-customization) without issue.

If this is indeed the case, then more than likely we won't be doing much here to alter this behavior - anything we introduced here would be an odd workaround to try and correct problems that one would be more likely happening in vSphere anyway, independent of Terraform. The correct course of action would be to determine what exactly is wrong with the VM that is keeping it from being customized properly, and correct those issues so the template is compatible with the customization process, or forgo customization for vApp properties in the case of OVAs, or DHCP and a provisioner in others.

Thanks!

[bill-rich]

Hey all,
I've done some work on reproducing this issue, and it does seem that the issue with the NIC not being connected is due to a failed customization on the virtual machine. As @jtdroste pointed out, the NIC is normally disconnected during the customization boot and then gets enabled after customization completes successfully. When the customization fails, the virtual machine will keep running and Terraform will report an error, but VMware will not reboot the virtual machine or connect the NIC.

Since this is the same behavior as when customization is performed directly through VMware, I think it would be best to avoid introducing the workaround in Terraform. When the issue comes up, it can be resolved by fixing the customization issue on the template. If the a virtual machine encounters customization errors, they will be logged to %WINDIR%\temp\vmware-imc on Windows or /var/log/vmware-imc/toolsDeployPkg.log on Linux. Hope this helps!

[emdantrim]

Hey all,

Sorry for beating a dead horse, but I was wondering -- do you think this is a problem with the OVA itself? I'm not entirely sure where to go from here, but I am tempted to open a bug report for the Ubuntu cloudimg OVA. Any further steps that you could suggest here would be absolutely stellar. ❤️

[vancluever]

Hey @emdantrim! I don't know if the problem would be with the OVA per se - as far as it's concerned, it's running "just fine" in that if you use it "as intended" (ie: configure through cloud-config/user data), then you'll be fine (as long as you can deal with the DHCP delay).

Unless you can find anything specific in the customization logs that would indicate a specific issue with the image that's causing customization to not work, then they might not do much about it 😕

What I would probably do: If you prefer customization over cloud-config (and with the Ubuntu OVA, I definitely wouldn't blame you 🙂), maybe just manually seed a template from a ISO install. This is what we do for our tests - you sidestep all of the weird config the OVA has with default DHCP and the serial console redirect, and cloud-init is not on the server when it's not necessarily needed. You can set the interfaces to a dummy static IP address and the boot-up/customization phase will take about 30 seconds or so.

Otherwise, if you are looking for an OVA that works really well with VMware and the vSphere provider, the best one I've found so far is the CoreOS OVA. You can configure the VM entirely from Ignition and pass that into the guestinfo.coreos.config.data property, and even generate it with the Ignition provider.

Hope this helps!

[ghost]

I'm facing this issue which is restricting me to do more automation. Has there been any fix on this? My cloned VM customization fails due to errors and if i check, the VM has its network interface disconnected.

[fenneh]

I too am experiencing the same issues.

I've tried different versions of the vSphere provider to no avail. Running vSphere 6.7.0.

If cloning from a Windows Template using customize the NIC doesn't appear to be connected. Sure, the customize may be failing.

In my case the customize is naming the machine and joining a domain - which works perfectly fine when running it via vSphere directly.

customize {
   network_interface = {}
   windows_options {
      computer_name = "${var.hostname}"
      join_domain = "${var.domain}"
      domain_admin_user = "${var.domainuser}"
      domain_admin_password = "${var.domainuserpw}"
      admin_password = "${var.secretpw}"
   }
}

From the cusomization logs

[2018-09-27T15:22:38:                              :  DEBUG] Temporary sysprep dir path: C:\sysprep
[2018-09-27T15:22:38:                              :  DEBUG] Copying file/directory from 'sysprep' to 'C:\sysprep'
[2018-09-27T15:22:39:                              :   INFO] Sysprep File is C:\Windows\Panther\unattend.xml
[2018-09-27T15:22:39:                              :   INFO] Password unobfuscation in progress
[2018-09-27T15:22:39:                              :  DEBUG] Decrypting passwords
[2018-09-27T15:22:39:                              :  DEBUG] Password found and decrypted for <AdministratorPassword><EncryptedValue>
[2018-09-27T15:22:39:                              :  DEBUG] Password found and decrypted for <Credentials><EncryptedPassword>
[2018-09-27T15:22:39:                              :  DEBUG] Tag <Value> not found after <Password>
[2018-09-27T15:22:39:                              :  DEBUG] Tag  -de  not found after <RunSynchronousCommand
[2018-09-27T15:22:39:                              :   INFO] Password unobfuscation successful
[2018-09-27T15:22:39:                              :  DEBUG] select * from win32_networkadapter where Manufacturer != 'Microsoft' and ServiceName != 'VMnetAdapter' and  manufacturer is not null and MACAddress is not null
[2018-09-27T15:22:39:                              :  DEBUG] Found 0 objects. Pointer 0. return code -2147217398(0x8004100a)
[2018-09-27T15:22:39:                              :  ERROR] WinMgmt : 
[2018-09-27T15:22:39:                              :  DEBUG] HandleCustomizationFailed
[2018-09-27T15:22:39:                              :  DEBUG] Customization in progress set to 0 at 2018-Sep-27 15:22:39
[2018-09-27T15:22:39:                              :   INFO] Cleaning up obfuscation details from registry
[2018-09-27T15:22:39:                 GuestCustUtil:  DEBUG] Status marker file C:/Windows/.post-gc-status doesn't exist
[2018-09-27T15:22:39:                              :  DEBUG] GOScPostRebootStatus feature is enabled
[2018-09-27T15:22:39:                 GuestCustUtil:  DEBUG] SetVmxProperty: key-'guestinfo.gc.notification', value-'WinMgmt : '
[2018-09-27T15:22:39:     WindowsOsAbstractionLayer:   INFO] VMware tools installation path=C:\Program Files\VMware\VMware Tools\
[2018-09-27T15:22:39:     WindowsOsAbstractionLayer:   INFO] Executing command: '"C:\Program Files\VMware\VMware Tools\vmtoolsd.exe" --cmd "info-set guestinfo.gc.notification WinMgmt : "'
[2018-09-27T15:22:39:     WindowsOsAbstractionLayer:   INFO] Command execution succeeded, output: '

The same issue has been reported here https://stackoverflow.com/questions/50482281/vspehere-vs-terraform-vm-customization-failure-with-network-not-being-connected

Update After spending a little time playing around. I figured out that the Template that I'd been provided had already been sysprep'd. After converting to a VM, starting, logging in, shutting down and converting it back to a template this appeared to fix any issues with customizations/NIC being enabled/disabled at correct times.

[riponbanik]

Guys has it been fixed to any new version or do we still need to manually turn the interface on?

I have used the settings -

wait_for_guest_net_timeout = 0
wait_for_guest_net_routable = false

This is the error i get for sysprepd machine -

[2018-10-25T02:50:16: : DEBUG] Copying file/directory from 'sysprep' to 'C:\sysprep'
[2018-10-25T02:50:16: : DEBUG] select * from win32_networkadapter where Manufacturer != 'Microsoft' and ServiceName != 'VMnetAdapter' and manufacturer is not null and MACAddress is not null
[2018-10-25T02:50:16: : DEBUG] Found 0 objects. Pointer 0. return code -2147217398(0x8004100a)
[2018-10-25T02:50:16: : DEBUG] Rpci: Sending request='deployPkg.update.state 4 101 C:/Windows/TEMP/vmware-imc/guestcust.log@WinMgmt : '

[2018-10-25T02:50:16: : DEBUG] Rpci: Sent request='deployPkg.update.state 4 101 C:/Windows/TEMP/vmware-imc/guestcust.log@WinMgmt : ', reply='', len=0, status=1

[2018-10-25T02:50:16: GuestCustUtil: DEBUG] Status marker file C:/Windows/.post-gc-status doesn't exist
[2018-10-25T02:50:16: : ERROR] error number 8004100a, WinMgmt :
[2018-10-25T02:50:16: : INFO] GuestCustUtil exiting.

[mlrtime]

I'm running into this issue with Centos 7.7, I'm simply trying to set a hostname and bring up a interface. What is odd is that once you customize, you need to provide a interface, whereas if you leave customization empty the nic comes up fine.

clone {
template_uuid = "${data.vsphere_virtual_machine.template.id}"
customize {
linux_options {
host_name = "vm-one"
domain = "example.com"
}
network_interface { }

}

}

Terraform Error:
Error:
Virtual machine customization failed on "/Lab/vm/TestC7":

timeout waiting for customization to complete

What is the workaround?

Terraform 0.12.12
provider.vsphere: version = "~> 1.13"
Centos 7.7
Vsphere 6.7

[k1ngpr4wn]

Also receiving the error @mlrtime mentions.

[uutest74]

have the same issue on windows templates with customize
Terraform 0.12.16
provider.vsphere v1.13.0
vSphere 6.5
Terraform say that apply was successful, and no errors in %WINDIR%\temp\vmware-imc
All network options applied by customization but network adapter not connected!
Why Terraform not tracking in state file connection of VM network adapter??? it is the on of most important parameters!
That issue ruin for me equal Terraform not for production usage with vSphere

[jaimehrubiks]

Same issue, but I think it is a problem with vsphere, not terraform. I'm not completely sure but for me it seems to be related to the template itself. Not sure why but I have a template with centos7 and network manager configured with dns and dhcp IP, the connection name is the one of the interface/device. When I create an instance from this template (terraform or GUI) it connects the network interface, for some others it seems to be random or disabled.

(vSpehre 6.7)

[jacobmammoliti]

I was having the same issue where my nic was showing up disconnected when I had a customize config.
I dug a bit and found the following error in /var/log/vmware-imc/toolsDeployPkg.log: Customization command failed: Failed to create bus connection: No such file or directory

I fixed this by adding After=dbus.service in my /lib/systemd/system/open-vm-tools.service and then re-cloning the template.

Re-running my terraform code with that updated template successfully fixed my error.

[pezhore]

@arctiqjacob's solution fixed my deployment woes.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!