Description
We need a way to expose some Oxide services to an external network. The most salient example is the customer providing us with some IP address where they'd like to reach the Nexus public API and web console. This address is not in our control, it's part of the customer network. To integrate this addressing with the Oxide rack network, we'll need to encapsulate this traffic onto the rack network, deliver it to the sled hosting that Nexus instance, where it'll be decapsulated for delivery to the actual program. In this comment, Robert proposed using OPTE for the on-sled part of this. The full path would start at the switch, where the customer network traffic is encapsulated into the rack physical network. OPTE is the complement to that, undoing the encapsulation on the way in and encapsulating the replies.
This issue tracks adding addresses for services like Nexus using this pattern. There are several pieces of work here, to be fleshed out.
- Adding an OPTE /
xdedevice. We currently only use this for guest traffic, so we'll have to extend things a bit to support other kinds of endpoints. - Create and manage system VNIs for this traffic. We currently reserve a range of VNIs for system and guest traffic, and we'll need to manage the system VNIs for this traffic.
- Potentially putting a VNIC over that. It's not clear if this will actually be needed. We do need it for guests currently, since the Viona driver that exposes a virtio-net device to guests only works over VNICs.
- An API for providing Nexus with this new, external address and asking it to listen there.
- There will be lots of work communicating this to the switch as well, such as the external address, VNI, and address of the physical sled to deliver the traffic to.