New Package Request: nmstate

[cathay4t]

https://github.com/nmstate/nmstate

1. What, if any, are the additional dependencies on the package? (i.e. does it pull in Python, Perl, etc)

   Both of them are rust based. Only depend on glibc.

2. What is the size of the package and its dependencies?

   nmstate-libs-2.1.0-0.3.alpha2.fc37.x86_64.rpm: 1.6MiB(1646494 bytes)
   nmstate-2.1.0-0.3.alpha2.fc37.x86_64.rpm: 2.0MiB (2140203 bytes)

3. What problem are you trying to solve with this package? Or what functionality does the package provide?

   Nmstate currently is used by OpenShift for host day 2 network configuration via kubernetes-nmstate through a container.
   Including nmstate into CoreOS could allow nmstate been used as day 1 and day 0 configuration where container is not allowed.
   The nmstate package ships with a CLI tool nmstatectl.
   The nmstate-libs pacakge ships C library for network configruation via NetworkManager. Openshift projects could use go binding of nmstate through this C library.

4. Can the software provided by the package be run from a container? Explain why or why not.

   Yes. But day1 and day0 has no container environments yet.

5. Can the tool(s) provided by the package be helpful in debugging container runtime issues?

   No. This is for host network configuration.

6. Can the tool(s) provided by the package be helpful in debugging networking issues?

   Yes. The nmstatectl show could provide current network states.

7. Is it possible to layer the package onto the base OS as a day 2 operation? Explain why or why not.

   No. This request is for enabling day 1 and day 0 network configurations.

8. In the case of packages providing services and binaries, can the packaging be adjusted to just deliver binaries?

   No service is provided by these two packages.

9. Can the tool(s) provided by the package be used to do things we’d rather users not be able to do in FCOS? (e.g. can it be abused as a Turing complete interpreter?)

   No.

10. Does the software provided by the package have a history of CVEs?

    No.

[travier]

We would need F36 packages to be able to include that. Could you link to the project repos & packages? Thanks!

[bgilbert]

What is the expected usage model for these packages? Would the Ignition config write an nmstate configuration and then a service to invoke nmstate to apply it?

[cgwalters]

I think all new package requests are going to be weighed with https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md in mind.

But...this one is about networking which may be needed for early boot.

I think honestly what I'd like to see is a signoff from the NetworkManager folks about a long term commitment to supporting this. Will this code be planned to ship in RHEL (9?) too?

[bgilbert]

I think all new package requests are going to be weighed with https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md in mind.

Yes, but that's strictly additive. There have been no discussions of ceasing support for the existing usage model.

[cathay4t]

@cgwalters This code is shipping in RHEL 8/9 with full support. I am the package maintainer in RHEL.

@thom311 as NetworkManager main devleoper, could you sign-off the long term support from NetworkManager on DBUS interface nmstate is using?

[cathay4t]

I will create Fedora 36/35/34 update for this before end of this week.

[thom311]

@thom311 as NetworkManager main devleoper, could you sign-off the long term support from NetworkManager on DBUS interface nmstate is using?

NetworkManager is not gonna break it's clients. In particular, it will stronlgy care to support nmstate.

[jlebon]

We started discussing this in this week's community meeting, but decided to punt until we could get some SMEs in attendance.

@cathay4t Would you and/or someone from the NMState team be able to join the Fedora CoreOS community meeting at some point? It is on Wednesdays at 16:30 UTC. @thom311 It would be good if you and/or someone from the NM team joined as well.

Feel free to reach out in #forum-coreos in Libera.Chat to coordinate.

[cathay4t]

Sure. Let me talk with openshift developers to get more detailed potential use cases before joining your meeting.

[cathay4t]

@jlebon I got 404 for link https://apps.fedoraproject.org/calendar/CoreOS which found at page of https://docs.fedoraproject.org/en-US/fedora-coreos/faq/ . Any more information for this Fedora CoreOS community meeting?

[bgilbert]

@cathay4t Looks like it's https://calendar.fedoraproject.org/CoreOS/ now. More info at https://github.com/coreos/fedora-coreos-tracker/#meetings.

[cathay4t]

Fedora 36 nmstate and nmstate-libs update is on-going at https://bodhi.fedoraproject.org/updates/FEDORA-2022-538995057e Fedoar 36 is shipping the updated version of nmstate and nmstate-libs

[dustymabe]

We discussed this in the community meeting today.

There was lots of thorough discussion. Some details that emerged:

• nmstate is useful for dynamically configuring a running system OR generating NM keyfiles with no running system needed.
• It accepts a yaml formatted file as input and this file represents the desired state of the entire system or a subset of the system.
• There are no plans to write a systemd unit that applies nmstate configuration on boot.
• The primary use case is applications that want to dynamically configure the Network state on the machine that don't want to do it via existing methods that already exist to do that (nmcli and/or dbus)
• A secondary use case is generating NM keyfiles to use for systems that are yet to be deployed

During the meeting we did come to an intermediate decision for the time being:

13:33:05   dustymabe   #agreed nmstate is useful for dynamically configuring
                       a running system OR generating NM keyfiles with no running
                       system needed. The nmstate YAML config itself can't be used
                       on system boot without an accompanying service to apply the
                       config. In order to use it for provisioning a user would need
                       to write a systemd unit themselves to apply the config they
                       wrote using Ignition. At this time we would like to continue
                       not including nmstate in the host and experiment with
                       applications that want to configure networking via nmstate
                       bundling it. 

However, we did agree to continue the discussion to further our understanding of the use cases of the hypothetical nmstate consumers and invited ffmancera and thaller to join us at our video community meeting next week.

[qinqon]

Including nmstate at FCOS will help with future effort of configuring networking at openshift installation for virtual and baremetal, right now for OVN the bridge is configured by a quite difficult to maintain 500 linux bash script [1] also at that point nmstate cannot being consumed as a container so it has to be on the nodes to integrate into that kind of scripits, but maybe the owners can share more about this use case @cybertron @jcaamano
[1] https://github.com/openshift/machine-config-operator/blob/48d88a7f75e97c87b2a6f4b00862a091fccbdd72/templates/common/_base/files/configure-ovs-network.yaml.

[zaneb]

I don't think CoreOS needs to provide a systemd service that reads and applies the nmstate yaml for nmstatectl to be useful. Users can write their own systemd service to do it and include it in the ignition. What they can't do is pull a container with nmstatectl in it to do that, because there may be network configuration required to access a container registry.

Currently this can be worked around by pre-generating the keyfiles using nmstatectl gc and shipping the keyfiles in the ignition. We're doing this in several places in OpenShift and it works, but it does limit where we can do it - currently it has to be in a container with the Python runtime and everything, so we can do it in e.g. the installer bootstrap host, but not in the installer itself. Now that the Rust library and its bindings are available, it's theoretically possible to embed this in a single binary, but the build process is not simple. Running nmstatectl gc remotely also has some limitation, like the fact that it can't actually verify that the (physical) interfaces referenced in the nmstate actually exist on the host.

In future it might be nice to make use of this in more integrated ways, during the initramfs - in particular it would be great to be able to apply a network config with ignition and then chain to another ignition file fetched over the network post-configuration. But including the binary at all is surely the first step.

[bgilbert]

I don't think CoreOS needs to provide a systemd service that reads and applies the nmstate yaml for nmstatectl to be useful. Users can write their own systemd service to do it and include it in the ignition.

They can, but that workflow seems unnecessarily awkward. If we expect it to be common, we should just ship the systemd unit.

Running nmstatectl gc remotely also has some limitation, like the fact that it can't actually verify that the (physical) interfaces referenced in the nmstate actually exist on the host.

Ah, that's an interesting detail. Would you consider it significantly better to generate the configs on the machine at runtime?

in particular it would be great to be able to apply a network config with ignition and then chain to another ignition file fetched over the network post-configuration.

Ignition fundamentally doesn't work that way. Configuration changes applied by Ignition only affect the real root when we switch into it, never the initrd environment where Ignition itself runs.

[cybertron]

Running nmstatectl gc remotely also has some limitation, like the fact that it can't actually verify that the (physical) interfaces referenced in the nmstate actually exist on the host.

Ah, that's an interesting detail. Would you consider it significantly better to generate the configs on the machine at runtime?

Yes. The ability to verify the configuration and roll it back if it's invalid is extremely helpful. Blindly writing nmconnection files works, but it makes for a very bad user experience if there is an error in the provided configs.

I'm also currently looking into an issue where the generated nmconnection files don't work, but if I apply the same config directly using nmstatectl then it does the right thing. That may be a bug in nmstate or it may just be a limitation of that configuration method, but either way it would be avoided if we could just run nmstatectl on the host.

in particular it would be great to be able to apply a network config with ignition and then chain to another ignition file fetched over the network post-configuration.

Ignition fundamentally doesn't work that way. Configuration changes applied by Ignition only affect the real root when we switch into it, never the initrd environment where Ignition itself runs.

This is a really big problem for using Ignition in complex (or even not so complex) network environments. Static IPs cannot be used. Bond modes that don't work degraded cannot be used. These are both common use cases in baremetal deployments.

This is a tangent to the NMState discussion, but I think we should give some thought to providing Ignition some way to configure networking before it attempts to retrieve configs served over the network.

[dustymabe]

This is a really big problem for using Ignition in complex (or even not so complex) network environments. Static IPs cannot be used. Bond modes that don't work degraded cannot be used. These are both common use cases in baremetal deployments.

We handle this just fine. See the docs and ask us any clarifying questions in the discussion forum.

This is a tangent to the NMState discussion, but I think we should give some thought to providing Ignition some way to configure networking before it attempts to retrieve configs served over the network.

Agree. This is a tangent. Let's keep this focused on NMState.

[cybertron]

I started a thread for the Ignition discussion so we don't clutter up this one: https://discussion.fedoraproject.org/t/network-config-needed-to-pull-ignition/39094

[qinqon]

Checking the ignition rationale, the verification step from nmstate align with that since it ensure that the network configuration state asked by the user is reached or fail, so it will fail to boot with a clear message about what was not able to setup.

There are scenarios that NetworkManager cannot verify that the configuration specify at a keyfile is reached:

• Setting MTU at some NICs doe not fail at NetworkManager but the outcome is not the expected one, It was related to SRIOV.
• Using linux-bridge vlan filtering with huge range vlans at ports fails when hardware offloading was configured at some type of NICs, this was only catched by nmstate verification
• Some ovs-bridge configurations fails when apply with keyfiles but works fine when apply at live system with nmstate https://bugzilla.redhat.com/show_bug.cgi?id=2088171
• For bonding there are some parameters convinations that only nmstate verification can catch:
  • The updelay options is only valid when miimon is bigger than 0.
  • The arp_ip_target require arp_interval bigger than 0
• There are some ethtool features mismatches that only verification can catch

In general there are a lot kernel restrictions we cannot code in user space but trust verification found it.

Apart from that running nmstatectl at live system allows to compose the network state with current network configuration with a tool like nmpolicyctl so instead of having some nmstate per node we can have just one per cluster and rendered by nmpolicyctl.

P.D: This was a message at the fedoraproject discussion, but it make more sense here and also editing the message does not alert the spam bot :-)

[bgilbert]

I worry that we've been talking past each other here, and I do think it would be useful to get this unblocked. I'll summarize my understanding of the situation and y'all can correct me.

Use cases

nmstate might be used by the administrator during initial provisioning of the node, or by the administrator to reconfigure the node at runtime, or by applications to update network configuration at runtime.

1. Application reconfiguration at runtime. Whenever possible, we do not ship software that is only useful for applications (or cluster orchestrators) at runtime, since such software should ship in the application container instead. Here, nmstate could run in a container and talk to the host NetworkManager via D-Bus.
2. Administrator reconfiguration at runtime. We generally discourage such reconfiguration. The administrator should reprovision the node instead.
3. Configuration during initial provisioning. This is potentially useful. In principle we could instruct users to render nmstate configs off-node and then copy in the resulting keyfiles, but they'd lose out on config validation that nmstate can perform at runtime. Therefore there's value in shipping nmstate in the OS, with a usage model where the user adds an nmstate config to the Ignition config.
   1. Non-autoconfig case. If the Ignition config isn't accessible until the network config is applied, there's a chicken-and-egg problem. On some platforms, Ignition needs network access to fetch the Ignition config at all. On all platforms, the Ignition config may reference arbitrary remote resources. And by design, any configuration written by Ignition — including network configuration — does not take effect until Ignition has finished running.
      1. Assumptions. Historically in addressing the chicken-and-egg problem, we've assumed that it only occurs on bare metal or simulated bare metal platforms (e.g. VMware), since cloud platforms will support network autoconfiguration such as DHCP. That is not always true (e.g. certain VPC configurations) but has been a workable simplification.
      2. CoreOS solution. CoreOS supports early network configuration by allowing users to embed network settings in the live ISO image used for bare metal installation. coreos-installer copies those settings onto the installed system, which applies them during the first boot before Ignition runs.
      3. Supporting nmstate. @qinqon is contributing customize: Add --network-nmstate option coreos-installer#864, which will allow embedding nmstate configs in the ISO by rendering them to NM keyfiles first. This is consistent with the current model, but loses nmstate's runtime validation features. We could go further and support such validation by allowing nmstate configs to be embedded directly in the live ISO, shipping nmstate in the initrd, and having it apply the configs before Ignition runs.
   2. Autoconfig case. Otherwise, we expect the node to be able to boot with basic autoconfigured networking (e.g. a management interface with DHCP), fetch the Ignition config and any referenced resources, apply the config, and continue booting. Any additional network configuration in the Ignition config (e.g. for bonded interfaces, or static IPs on additional interfaces) is then activated later in boot, after Ignition completes.
      1. Supporting nmstate. It seems natural to ship nmstate on the host, with a systemd service that automatically reads files out of a directory (e.g. /etc/nmstate) and applies them. The Ignition config could then simply write out nmstate files, just as it can with NM keyfiles today. If the nmstate service doesn't persist the network config into NetworkManager, it could run on every boot; if it does persist the config, it could run only on the first boot, or delete the files from /etc/nmstate after they've been applied.

Problem statement

Overall this seems like a reasonably straightforward change: ship nmstate and let its systemd service apply user configuration at boot, and maybe eventually add extra initrd support for the non-autoconfig case. And that leaves me very confused when I hear that nmstate has no plans to ship such a systemd service. If nmstate is only meant to be used by applications to do reconfiguration at runtime, then we don't need to ship it in the OS. If it's meant to be useful for initial node configuration, then users will need a way to apply nmstate configs.

What am I missing?

[cathay4t]

I(Nmstate developer) like the idea of nmstate shipping systemd service applying network state from /etc/nmstate. Let me talk to openshift-installer, coreos-installer and openshift baremetal installer team about this.

[qinqon]

@bgilbert about chicken-egg issue, we can do both things, keep the option to inject keyfiles generated by nmstate so initial networking is propertly configured to access the iginition url and also the systemd nmstate service to apply nmstate config from that retrieved ignition from the url, does it make sense ?

So users will have small initial nmstate configuration that is not validated and just allow to access ignition url and proper nmstate to setup the cluster that will be validated.

At a very naif use case for static non-dhcp interfaces, users will set the IPs with this initial nmstate so they can retrieve the ignition from the url and the ignition will contain the full nmstate to setup stuff like bridges, bonds and the like and it will be verified.

I understand that injecting initial nmstate can only be done for baremetal or psudo baremetal where we can customize the images right ? I don't now if we have scenarios with cloud + non-dhcp.

[bgilbert]

@qinqon Yes, I agree.

I understand that injecting initial nmstate can only be done for baremetal or psudo baremetal where we can customize the images right ? I don't now if we have scenarios with cloud + non-dhcp.

Right. Specifically, it can only be done when installing with coreos-installer, and not when launching directly from an OS image.

[qinqon]

@bgilbert related to "problem statement"

Overall this seems like a reasonably straightforward change: ship nmstate and let its systemd service apply user configuration at boot, and maybe eventually add extra initrd support for the non-autoconfig case. And that leaves me very confused when I hear that nmstate has no plans to ship such a systemd service. If nmstate is only meant to be used by applications to do reconfiguration at runtime, then we don't need to ship it in the OS. If it's meant to be useful for initial node configuration, then users will need a way to apply nmstate configs.

Since Nmstate is happy to add the systemd mechanism we are all set here right ? we are covering how the user will put initial nmstate configuration. Also I understand that the install nmstate config file to the system is already covered by normal ignition config so it does not need any integration with coreos-installer.

Since looks like we have green light here I think these are the the next steps, right ?

• Add systemd setup at nmstate
• Release nmstate crate/rpm to fedora
• Create a PR at https://github.com/coreos/fedora-coreos-config

And in parallel review/merge the coreos-installer --nmstate-network PR.

[travier]

Should we close this one now that coreos/fedora-coreos-config#2269 is merged?

[bgilbert]

No, the remaining steps are in #1175 (comment).

[dustymabe]

The fix for this went into next stream release 38.20230322.1.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 37.20230322.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 37.20230322.3.0.

[dustymabe]

This is now in all streams of FCOS. Do we have anyone who is going to work on the remaining steps in #1175 (comment) ?

cc @qinqon @bgilbert

[bgilbert]

@qinqon, would you be able to write up a documentation PR for using nmstate in Fedora CoreOS?

[qinqon]

@qinqon, would you be able to write up a documentation PR for using nmstate in Fedora CoreOS?

Sure no problem

[qinqon]

@bgilbert @dustymabe I have start document it here:

• network: Add nmstate examples fedora-coreos-docs#531

There is a missing feature at nmstate to do the same that is done at those examples:

• https://bugzilla.redhat.com/show_bug.cgi?id=2187622

Also the fcos examples looks a little strange since they are doing static IP configuration but they set dhcp-hostname.

[dustymabe]

We discussed this in the community meeting today.

12:48:06 dustymabe | #agreed we will document a single or a few examples with nmstate
                   | (perhaps putting them on a secondary page and linking from the
                   | main page) and leave keyfiles as the primary documented network
                   | configuration for now.