Support replacing multicast bootstrap with local router API

[rcgoodfellow]

Background

The initial implementation of cluster bootstrap was built using multicast. However, we do not anticipate supporting anything beyond link-local multicast on the sidecar router, and each sidecar port is an isolated L2 broadcast domain. This precludes sleds from discovering each other using link-local multicast. Thus, an alternative mechanism is needed for the Oxide control plane to discover compute sleds during bootstrap. Further background can be found in Section 6.3 of RFD 63

Progress

A first cut of a local router API that supports advertising and querying compute sled underlay addresses has been added to the DDM API and is usable from ddm-local. See the ddm-local README for details on what this looks like.

[smklein]

Regarding the plan for network bootstrap via Sled-unique IPv6 prefixes via the local routing daemon: sounds great. I read through section 6.3 of RFD 63, and that all makes sense (also makes sense why the link-local limitation of multicast makes the current implementation in Omicron Not Great).

Reading through the docs for DDM Local, there are a couple things which stand out to me as questions regarding integration - I'm not sure precisely how we should pick this up and plug it into Omicron to replace the existing implementation...

Expectations re: Running DDM?

DDM local does not modify any underlying routing table state, it is strictly for testing the protocol plane, and external integration with DDM. DDM local works by setting up and connecting simnet links between routers.
...
With ddm-local, all of this happens on a single server for ease of testing.

If DDM is dependent on setting up simnet links, is the expectation that "we would need to run multiple copies of the control plane software on a single physical sled" to get this working? Or are you expecting that we'd be able to run the ddm stack on two separate sleds in the lab environment?

As a heads-up, the sled agent currently expects that it is doing sled-wide resource management (of zones, ZFS filesystems, etc) so it isn't really possible to run multiple instantiations of the sled agent on a single sled right now.

API to DDM

The API to access DDM - in particular, advertise_prefix and get_peers - look like a great approach for the bootstrap code in Omicron. Seems like these should eventually allow a 1:1 replacement of the IPv6 multicast code.

That being said, from my reading of the README in https://github.com/oxidecomputer/netadm-sys , I kinda expected that these functions would be provided over a library-based API. Is the software managing the routing stack on a Gimlet going to be running in a separate service? If yes, will this service be initialized before the Sled Agent? How should the Sled agent know how to contact it?

[rcgoodfellow]

If DDM is dependent on setting up simnet links, is the expectation that "we would need to run multiple copies of the control plane software on a single physical sled" to get this working? Or are you expecting that we'd be able to run the ddm stack on two separate sleds in the lab environment?

As a heads-up, the sled agent currently expects that it is doing sled-wide resource management (of zones, ZFS filesystems, etc) so it isn't really possible to run multiple instantiations of the sled agent on a single sled right now.

tl;dr; - For ddm-local, yes, the expectation is everyone is running on the same host. However, for multiple hosts, we can just use ddm-illumos which is what we'll be using in production.

The distinction between these two things is what's called the platform implementation in Maghemite. In Maghemite, routing protocols are built in terms of a protocol implementation and a platform implementation. The protocol implementation contains the core logic of a routing protocol and is implemented in terms of a platform trait. A platform trait implementation provides the services from an underlying platform that are needed to implement a routing protocol, such as implementing messaging interfaces through network sockets, adding routing table entries to an OS kernel or an P4 table, etc.

ddm-local is different from ddm-illumos in the sense that ddm-illumos looks for the real system interfaces it should use for peering, ddm-local creates and connects simnet interfaces to realize a mock physical topology locally. API operations between the two are the same, the only differences are in the implementation details.

The API to access DDM - in particular, advertise_prefix and get_peers - look like a great approach for the bootstrap code in Omicron. Seems like these should eventually allow a 1:1 replacement of the IPv6 multicast code.

I think the replacement will be advertise_prefix and get_prefixes. Peers are only directly connected neighbors, so sleds will never see each other through peers except for the sidecar connected sled, who will see every sled in the rack as a peer. Prefixes, on the other hand, will propagate through transit routers and be visible to all sleds.

That being said, from my reading of the README in https://github.com/oxidecomputer/netadm-sys , I kinda expected that these functions would be provided over a library-based API. Is the software managing the routing stack on a Gimlet going to be running in a separate service? If yes, will this service be initialized before the Sled Agent? How should the Sled agent know how to contact it?

So there is a library-based API that's created by progenitor, but it's making API calls under the hood.

Yes, the router is an active daemon. Netadm-sys is a crate we use for interacting with the network subsystems on illumos. It's used by various platform implementations in Maghemite. I don't think the routing daemon will depend on the sled agent, so we can likely have it run before the sled agent (maybe have this as an SMF dependency?). When the routing daemon is started, we can tell it what address and port to listen on - so this can be in a config file somewhere that the sled agent picks up and can also have a well-known default like localhost:8047.

[rcgoodfellow]

Closing this as the admin APIs are here to support this.