Network issues in machines with two networks

[jrpedrianes]

Reviewing the current network configuration I get some "warnings" when I want to apply it again:

root@lb-01:~# netplan apply 

** (generate:2128): WARNING **: 09:49:24.457: `gateway4` has been deprecated, use default routes instead.
See the 'Default routes' section of the documentation for more details.

** (generate:2128): WARNING **: 09:49:24.457: `gateway4` has been deprecated, use default routes instead.
See the 'Default routes' section of the documentation for more details.

** (generate:2128): WARNING **: 09:49:24.457: Problem encountered while validating default route consistency.Please set up multiple routing tables and use `routing-policy` instead.
Error: Conflicting default route declarations for IPv4 (table: main, metric: default), first declared in ens3 but also in ens4

Also if I check the current "IP routes" applied to the machine, I get that has configured more than one default route, in my case I have 3 :

root@lb-01:~# ip route
default via 10.10.10.1 dev ens3 proto static onlink 
default via 192.168.1.1 dev ens4 proto static 
default via 10.10.10.1 dev ens3 proto dhcp src 10.10.10.10 metric 100 
10.10.10.0/24 dev ens3 proto kernel scope link src 10.10.10.10 metric 100 
10.10.10.1 dev ens3 proto dhcp scope link src 10.10.10.10 metric 100 
192.168.1.0/24 dev ens4 proto kernel scope link src 192.168.1.210 

Reviewing the applied configuration, which you can see next, I saw that you are using deprecated configurations and also you are not defining any default route:

network:
    ethernets:
        ens3:
            dhcp4: true
            gateway4: 10.10.10.1
            nameservers:
                addresses:
                - 4.4.4.4
                - 8.8.8.8
                search:
                - virnat0.nat.local
        ens4:
            addresses:
            - 192.168.1.210/24
            dhcp4: false
            gateway4: 192.168.1.1
            nameservers:
                addresses:
                - 4.4.4.4
                - 8.8.8.8
                search:
                - external0.macvtap.local
    version: 2

Seems that these configurations: dhcp4: true, gateway4: 10.10.10.1 and gateway4: 192.168.1.1 are the culprits that appear more than one default route.

Maybe is better don't use DHCP, pin the IP to the network and define a "default" route, as follows:

network:
    ethernets:
        ens3:
            addresses:
            - 10.10.10.20/24
            nameservers:
                addresses:
                - 4.4.4.4
                - 8.8.8.8
                search:
                - virnat0.nat.local
        ens4:
            addresses:
            - 192.168.1.210/24
            nameservers:
                addresses:
                - 4.4.4.4
                - 8.8.8.8
                search:
                - external0.macvtap.local
            routes:
                - to: default
                  via: 192.168.1.1
    version: 2

In this commit you can see all the modifications that I have applied to the metal-could project to get this network configuration:

jrpedrianes@b95b82d

In summary:

• Don't use DHCP, so we configure libvirt_network to doesn't start up the DHCP service.
• Pin the IP to the network, so we don't attach the IP in the libvirt_domain terraform configuration.
• Mark one network interface as default to add the default route only to one network configuration.

[jrpedrianes]

@achetronic I have updated (locally) the terraform code to fix this using the latest commits.

Do you want that I create a PR with the changes, or prefer to discuss first the solution?

[jrpedrianes]

Hi @achetronic, I pushed all the modifications that I did to try to address this issue in case you want to look at it first before doing the PR, you can check it here: main...jrpedrianes:metal-cloud:main

[achetronic]

Hello @jrpedrianes let's discuss the topic. By the moment DHCP is automatically used on NAT networks and fixed IPs on macvtap. This was done because completely needed when I coded it long time ago.

I don't like deleting features, and your initial proposal was fixing the IPs always. Honestly I think you can be right on that and may be we should fix them all

WDYT about this?

(sorry for the delay in advance, several meals in the ovel at this moment)

[jrpedrianes]

I think that in this case, the use of DHCP does not add much since from the beginning you have to configure the machine IP, if you didn’t have to configure it we should use DHCP and I would try to find a way to fix it keeping it, but now not.

Also, as you configure the external IP statically I think that is better do the same in both cases, easier to review and understand how the networking is configured.

[achetronic]

I think that in this case, the use of DHCP does not add much since from the beginning you have to configure the machine IP, if you didn’t have to configure it we should use DHCP and I would try to find a way to fix it keeping it, but now not.

Also, as you configure the external IP statically I think that is better do the same in both cases, easier to review and understand how the networking is configured.

Well, I think it's a fair point and explanation

Let's do this. Can you open a PR with the proposal? but please, don't execute Terraform fmt on this iteration, just the code changes to make the review easier, as they are not trivial changes and we don't wanna break this for the users. Let's execute the fmt in another PR, right?

The idea is fixing the IPs for all the machines, and drop DHCP support. Let's implement DHCP in a better way somewhen in the future

We would need to support IPv4 ips, and may be IPv6. but we need validation for them too

[jrpedrianes]

Ok, let me remove all the fmt stuff and the I will open a PR

[achetronic]

Released here: https://github.com/achetronic/metal-cloud/releases/tag/v1.1.0

Thank you for all the effort put on this :)