unable to create a static ipv6 routed domain

[lgaggini]

Version Reports:

Distro version of host:

CentOS Linux release 7.5.1804 (Core)

Terraform Version Report

Terraform v0.11.8

Libvirt version

3.9.0

terraform-provider-libvirt plugin version (git-hash)

f104fad4

---

Description of Issue/Question

Setup

provider "libvirt" {
    uri = "qemu:///system"
}

resource "libvirt_network" "public"{
  name = "public"
  mode = "route"
  bridge = "br-public"
  addresses = ["{{ ipv6_network }}"]
  dhcp {
    enabled = "false"
  }
  autostart = "true"
}

resource "libvirt_domain" "tf_test" {
  # init
  name = "tf_test"
  metadata = "tf_test"
  vcpu = 2
  memory = 512
  running = false
  autostart = false

  # network
  network_interface {
    network_id = "${libvirt_network.public.id}"
    addresses = ["{{ ipv6_address }}"]
    # wait_for_lease = true
  }
}

Steps to Reproduce Issue

terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + libvirt_domain.tf_test
      id:                               <computed>
      arch:                             <computed>
      autostart:                        "false"
      emulator:                         <computed>
      machine:                          <computed>
      memory:                           "512"
      metadata:                         "tf_test"
      name:                             "tf_test"
      network_interface.#:              "1"
      network_interface.0.addresses.#:  "1"
      network_interface.0.addresses.0:  "{{ ipv6_address }}"
      network_interface.0.hostname:     <computed>
      network_interface.0.mac:          <computed>
      network_interface.0.network_id:   "${libvirt_network.public.id}"
      network_interface.0.network_name: <computed>
      qemu_agent:                       "false"
      running:                          "false"
      vcpu:                             "2"

  + libvirt_network.public
      id:                               <computed>
      addresses.#:                      "1"
      addresses.0:                      "{{ ipv6_network }}"
      autostart:                        "true"
      bridge:                           "br-public"
      dhcp.#:                           "1"
      dhcp.0.enabled:                   "false"
      mode:                             "route"
      name:                             "public"


Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes
  
  libvirt_network.public: Creating...
  addresses.#:    "" => "1"
  addresses.0:    "" => "{{ ipv6_network }}"
  autostart:      "" => "true"
  bridge:         "" => "br-public"
  dhcp.#:         "" => "1"
  dhcp.0.enabled: "" => "false"
  mode:           "" => "route"
  name:           "" => "public"
libvirt_network.public: Creation complete after 5s (ID: 3cc28126-de07-4cbc-b4ff-cd62c234cc38)
libvirt_domain.tf_test: Creating...
  arch:                             "" => "<computed>"
  autostart:                        "" => "false"
  emulator:                         "" => "<computed>"
  machine:                          "" => "<computed>"
  memory:                           "" => "512"
  metadata:                         "" => "tf_test"
  name:                             "" => "tf_test"
  network_interface.#:              "" => "1"
  network_interface.0.addresses.#:  "" => "1"
  network_interface.0.addresses.0:  "" => "{{ ipv6_address }}"
  network_interface.0.hostname:     "" => "<computed>"
  network_interface.0.mac:          "" => "<computed>"
  network_interface.0.network_id:   "" => "3cc28126-de07-4cbc-b4ff-cd62c234cc38"
  network_interface.0.network_name: "" => "<computed>"
  qemu_agent:                       "" => "false"
  running:                          "" => "false"
  vcpu:                             "" => "2"

Error: Error applying plan:

1 error(s) occurred:

* libvirt_domain.tf_test: 1 error(s) occurred:

* libvirt_domain.tf_test: virError(Code=27, Domain=19, Message='XML error: Invalid to specify MAC address '76:92:e8:5a:a4:9b' in network 'public' IPv6 static host definition')

If I try to uncomment wait_for_lease = true the domain is correctly created, terraform doesn't end because dchp enabled is false, so no lease will be assigned to the domain

If I choose a custom (not computed) MAC address I get the same error. Maybe I'm missing some obvious.

I attach a trace level terraform log:
tf_test_static_ipv6_routed.log

Feel free to ask any additional information and keep up the good work on this nice project.

[lgaggini]

Ciao,
before to post logs and stdout I replaced my ipv6 network and ipv6 address with placeholders using jinja2 syntax to hide them. I wasn't clear about that in the first post, sorry, my fault.
I experience the problem with some real word ipv6 network and address. I didn't try an ipv4 routed domain like in your example, I'll try and let you know but I suspect it's a ipv6 only problem.

[lgaggini]

Ok, like you recreate, using an ipv4 addressing there is no problem:

provider "libvirt" {
    uri = "qemu:///system"
}

resource "libvirt_network" "public4"{
  name = "public4"
  mode = "route"
  bridge = "br-public4"
  addresses = ["10.0.1.0/24"]
  dhcp {
    enabled = "false"
  }
  autostart = "true"
}

resource "libvirt_domain" "tf_test4" {
  # init
  name = "tf_test"
  metadata = "tf_test"
  vcpu = 2
  memory = 512
  running = false
  autostart = false

  network_interface {
    network_id = "${libvirt_network.public4.id}"
    addresses = ["10.0.1.3"]
    # wait_for_lease = true
  }
}

but using ipv6 addressing:

resource "libvirt_network" "public6"{
  name = "public6"
  mode = "route"
  bridge = "br-public6"
  addresses = ["2a01:4f8:212:2122::/64"]
  dhcp {
    enabled = "false"
  }
  autostart = "true"
}

resource "libvirt_domain" "tf_test6" {
  # init
  name = "tf_test"
  metadata = "tf_test"
  vcpu = 2
  memory = 512
  running = false
  autostart = false

  network_interface {
    network_id = "${libvirt_network.public6.id}"
    addresses = ["2a01:4f8:212:2122::3"]
    # wait_for_lease = true
  }
}

the error appears:

libvirt_domain.tf_test6: virError(Code=27, Domain=19, Message='XML error: Invalid to specify MAC address 'ae:3c:38:4d:88:c6' in network 'public6' IPv6 static host definition')

[MalloZup]

@lgaggini ok thx! i will have a look then

[MalloZup]

@lgaggini update sofar:

i checked the code in thi repo and everything is ok.
To me look more a libvirt issue/missing feature upstream.

i could look at libvirt upstream : https://github.com/libvirt/libvirt/blob/7564daca8a71dfec4a22c55b6f11f266f7044e05/src/conf/network_conf.c#L411

Would maybe interesting if you can reproduce the same issue with virsh ( i think the provider codebase here is fine)

[lgaggini]

Thank you.
I'll do some investigation on libvirt too,I'll try to reproduce the same issue on virsh and I'll let you know.

[lgaggini]

Ok, I think i've figured out what the problem is.

If an ipv6 network / addressing scheme is used, libvirt checks if in the input xml for network manipulation there is the field mac and if it's present gives the error:

Invalid to specify MAC address 'XX:XX:XX:XX:XX:XX' in network 'public' IPv6 static host definition'

This because for the ipv6 the use of mac address (in a dchp context, like used by libvirt) is deprecated and replaced by DHCP Unique Identifier (DUID).

The provider doesn't seem to make difference between ipv4 and ipv6 and it always generates a mac address.

Then the generated mac address in the libvirt/utils_libvirt_go is used to create the xml which is passed to libvirt-go, processed by libvirt-go
and at the end arrives at libvirt where the error pops up.

[MalloZup]

@lgaggini yop make perfect sense, i just tried out your suggestion, i just gave the xml definition an empty mac and network was up.

Thank for sharing the infos about the DUID and DHCP:+1: This was really awesome :cupid:
now i am asking myself and reading if we should for all network types not generate an IPV6 mac, and more generally if we should generate MACs,

From https://avdv.github.io/libvirt/formatnetwork.html i read

---

Generally it is best to not specify a MAC address when creating a network - in this case, if a defined MAC address is needed for proper operation, libvirt will automatically generate a random MAC address and save it in the config. Allowing libvirt to generate the MAC address will assure that it is compatible with the idiosyncrasies of the platform where libvirt is running. Since 0.8.8
dns

---

So i am thinking that we should not generate them (on ipv4 and ipv6) . I will try out some tests and if ok i will remove the mac generation.

[MalloZup]

@lgaggini can you do some tests from the upstream branch linked and give me some feedback? this would be awesome.

I already did some tests are ok, but just to be on safer side.

for doing the test

you need to build from source and then move the binary to /usr/bin
https://github.com/dmacvicar/terraform-provider-libvirt#building-from-source
otherwise is ok, i plan to merge this tomorrow afternoon more or less

[lgaggini]

Sure, I'll do some tests on my use case and I'll let you know.

[MalloZup]

@lgaggini thx! atm i am fixing wait_for_lease =true this will not work. If you find other issue let me know.. 🏩

[lgaggini]

Ok, using your feature branch nomac I'm correctly able to create ipv6 routed domain.

This:

resource "libvirt_network" "public6"{
  name = "public6"
  mode = "route"
  bridge = "br-public6"
  addresses = ["2a01:4f8:212:2122::/64"]
  dhcp {
    enabled = "false"
  }
  autostart = "true"
}

resource "libvirt_domain" "tf_test6" {
  # init
  name = "tf_test"
  metadata = "tf_test"
  vcpu = 2
  memory = 512
  running = false
  autostart = false

  network_interface {
    network_id = "${libvirt_network.public6.id}"
    addresses = ["2a01:4f8:212:2122::3"]
  }
}

now works withour error.

[MalloZup]

yop this works nice, atm the wait_for_lease to true we don"t get the IP. feel free to experiment other corner cases.. thank you much for all efforts and idea sharing 👍

[MalloZup]

@lgaggini atm i am little stuck on this bug because removing the MAC in networkdefinition we get other regression .. 🎪 😄 .. especially when using wait_for_lease.

I have asked to @dmacvicar for some infos, maybe he has more insights. Otherwise i need to move more in deep in that feature, the networking part of the provider is kind of complex, so it will take me more time to analyze the whole picture
Thx for you patience 👍 and insights! if you feel that you can fix it, take inspiration from my PR ! this would be also awesome 😃

[zeenix]

@MalloZup This one seems to be waiting on you to finish your PR. :) BTW, please add some more info to your git commit messages themselves. One doesn't always have internet (think flights) access and details in git history are your only hope then to figure how why some change was introduced. :)

[frgomes]

I've stumbled with this bug too :-(

A trick to circumvent this bug

1. Employ a pre-existing routed network, purely IPv6. Something like this:

<network ipv6='yes'>
  <name>babe.network</name>
  <forward dev='enp5s0' mode='route'>
    <interface dev='enp5s0'/>
  </forward>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:ba:be:00'/>
  <domain name='babe.example.com'/>
  <ip family='ipv6' address='2001:470:cafe:cafe::1' prefix='64'/>
</network>

2. On your network interface configuration, DO NOT specify any MAC address and ignore any leasing requirement. Libvirt does not support MAC addresses on IPV6 configurations, so do not provide any. You are going to configure static IP addresses, so there's nothing to be leased. And, the most important: DO NOT specify any addresses for IPv6 interfaces, since this is one of the root causes for this bug. For example, provide this:

  network_interface {
    network_name   = "babe.network"
    hostname       = "master-node-0
  }

3. Make sure that your configuration.tf file DOES NOT return any IP addresses. You don't need that, since you know beforehand which IP addresses you are going to configure. But, if you find convenient that Terraform returns IP addresses to the shell, devise a way to hardcode the return values or populate an array somehow with the addresses you know beforehand should be there. I have commented out the output block, since I don't need it:

# You don't need this block below
# output "ipv6" {
#   value = [
#     libvirt_domain.masters.*.network_interface.0.addresses,
#     libvirt_domain.workers.*.network_interface.0.addresses
#   ]
# }

4. Now inject a configuration script via cloud-init or ignition or similar. In your script, you will configure static IPv6 addresses. This below is an example utilizing ignition, since I'm deploying OpenSUSE MicroOS:

{
  "ignition": {
    "version": "3.2.0"
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "path": "/etc/hostname",
        "mode": 420,
        "overwrite": true,
        "contents": {
          "source": "data:,${hostname}"
        }
      },
      {
        "filesystem": "root",
        "path": "/etc/sysconfig/network/ifcfg-eth0",
        "mode": 420,
        "overwrite": true,
        "contents": {
          "source": "data:,BOOTPROTO%3Dstatic%0ASTARTMODE%3Dauto%0AIPADDR_0%3D${network_prefix}::${cluster}:${node}%0ADNS1%3D${network_dns1}%0ADNS2%3D${network_dns2}%0A"
        }
      },
      {
        "filesystem": "root",
        "path": "/etc/sysconfig/network/routes",
        "mode": 420,
        "overwrite": true,
        "contents": {
          "source": "data:,default%20${network_gateway}%20-%20-%0A"
        }
      }
    ]
  }
}

---

These are the files created in the virtual machine, which runs OpenSUSE MicroOS:

/etc/hostname

master-node-0

/etc/sysconfig/network/ifcfg-eth0

BOOTPROTO=static
STARTMODE=auto
IPADDR_0=2001:470:cafe:cafe::babe:32
DNS1=2606:4700:4700::1111
DNS2=2606:4700:4700::1001

/etc/sysconfig/network/routes

default 2001:470:cafe:cafe::1 - -

---

Notes

• Maybe you could configure Terraform for creating the routed network for you, instead of doing it by hand via virsh. I was not brave enough to try this alternative after an entire day wasted with this bug.

References

• Ignition syntax specification