Centos7: Vmware ethernet name changes when booted in vagrant with persistent udev naming rules

[jameskyle]

I'm creating vagrant box's with packer. The process goes smoothly without error.

After the box is imported and a vagrant up is called, the resulting image has a different interface name.

The name is of the following format (as is the one in packer packer, just a different name): eno16777728

CentOS 7 uses persistent udev naming rules. The above naming format indicates it's using the bios device index. For the name to change, the bios index would need to change.

I assume this is a result of a change in the vmx file resulting in a change in device enumeration by the bios.

[jameskyle]

Comparing the box's vmx and the vagrant images vmx, it's changing the

ethernet0.pciSlotNumber

This would change the persistent naming result.

[tnarik]

Something similar happens for Solaris boxes. I have a guest box which is created via packer and sets e1000g0 as the network interface. The resulting vmx file has a specific ethernet0.pciSlotNumber = "33" and attempting to boot it directly succeeds, with network connectivity.

If the guest box is cloned via vagrant with a no configured Vagrantfile (only parameter is config.vm.box = "tnarik/solaris10"), then the ethernet0.pciSlotNumber value of the resulting vmx file changes (to "32"). When logging to the guest box, I can see a 'e1000g1' interface, but not trace of 'e1000g0'. vagrant ssh doesn't work in this case.

If the Vagrantfile is configured to force the original value (the one set by the packer build, so "33"), then the 'e1000g0' network interface is properly configured during startup (because it exists), there is no trace of 'e1000g1' (which we didn't configure), and vagrant ssh succeeds.

[tnarik]

In packer I see the pciSlotNumber is hardcoded, as per https://github.com/mitchellh/packer/blob/7ad8f33ddb66e4bd1cfbf8c41a62de2c026bbf7d/builder/vmware/iso/step_create_vmx.go.

Obviously I cannot do the same thing for the plugin, but I have the feeling it is the same approach (just with a different number), or it assumes that moving PCI slots around would be ok (which is not true for some OS).

[lattwood]

Just ran into this after purchasing the vmware provider, sent an email to support@hashicorp.com (it'll come from jnickel.com, @mitchellh )

[cbednarski]

It might be VMware doing this. When you import or clone a VM via the GUI, VMware asks you whether you want to reinitialize the network interfaces. The reason is that these interfaces are associated with additional data like VMnet IPs and MAC addresses in VMware's virtual networking layer. If you have duplicate MACs or IPs set the boxes won't be able to connect to the network (or maybe won't boot).

I presume for simplicity and "just works" functionality, Vagrant (or possibly VMware) assigns a new NIC when you bring up the box. You'll also get a new one when you destroy and up the box. Without this, you would likely need to create a separate vmnet VLAN for each VM so it can have its own network segment which doesn't conflict with other VMs.

I think as a work-around for this you can use the Vagrant provider section to write the vmx data for the interface exactly the way you want, but you'll have to dig into the vmx format a bit. The easiest way I've found to do this is to copy the data from an existing vmx file that's already configured the way I want.

Something like:

config.vm.provider "vmware_fusion" do |v|
  v.vmx["ethernet1.pciSlotNumber"] = "32"
  # etc.
end

In the clone scenario I mentioned, the GUI does provide a way to keep the same interface if that's what you want. There may be something we can pass to the VMware provider to use this behavior with Vagrant and expose via a setting like reinitialize_nic: false, but it would be good to confirm this will work.

I think doing this may break vagrant in subtle ways, since you won't be able to up more than one machine from the same Vagrantfile. Also, not an expert and I didn't test this, so I might be completely wrong. :)

/cc @sethvargo @lattwood

[lattwood]

Thanks for the speedy response

It seems like VMWare's behaviour when adding an additional virtual network adapter is deterministic enough that copying the configuration for ens33 to ens32 during the packer build process works around this behaviour. Since I'm creating multiple VMs from the vagrantfile the VM workaround is the appropriate solution for this case.

[jameskyle]

@cbednarski

It's true that creating a new nic is a good policy to avoid MAC collisions and such. However, this particular problem maps to which virtual slot the card is plugged into.

The location the card is plugged into should have no effect on network configuration or conflicts.

[mitchellh]

I recommend going with @lattwood's workaround unless we can come up with a way for Vagrant to do this automatically .Thanks!

[rmoriz]

It seems to me this also affects/breaks Ubuntu 15.10 + 16.04 server released with systemd-based predictable interface naming (afaik the distro default, which is based on the hotplug slot for external network interfaces (non-onboard) like VMWare simulates, see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ )

[rmoriz]

Could it be that VMWare recently changed the default NIC slot ID from 33 to 32 so packer could just do the change, too? When manually setting the VMX value on build time to 32 I have no problems with vagrant anymore in later clones/uses. See chef/bento#545 (comment)