Ingest ereports from SPs #370

hawkw · 2025-03-27T17:41:07Z

This pull request implements the MGS side of the SP ereport ingestion
protocol. For more information on the ereport ingestion protocol, refer
to the following RFDs:

In particular, this branch makes the following changes:

Add types to gateway-messages representing the ereport protocol wire
messages exchanged between MGS and the SP; these are defined in
RFD 545.
Somewhat substantial refactoring to the shared_socket module in
gateway-sp-comms. Currently, the SharedSocket code for handling
received packets is tightly coupled to the control plane agent message
types. Ereport requests and responses are sent on a separate UDP port.
Therefore, I've hacked up this code a bit to allow SharedSocket to
be generic over a RecvHandler trait that defines how to handle
received packets and dispatch them to single-SP handlers. This is
implemented for both the control-plane-agent protocol and, separately,
for the ereport protocol.
Actually add said implementation of the ereport protocol, including
code for decoding ereport packets and a per-SP worker task that tracks
the metadata sent by the SP and adds it to each batch of ereports.

A corresponding Omicron branch, oxidecomputer/omicron#7903, depends on
this branch and integrates the ereport code into the MGS app binary and
the SP simulator.

This commit refactors `gateway_sp_comms::SharedSocket` to make the received message handler a generic trait. This way, the `SharedSocket` type and its associated machinery for discovering SPs and forwarding received messages to per-SP handlers can be used for the ereport ingestion socket as well as for `control-plane-agent` messages.

oopsie

hawkw · 2025-04-07T16:34:42Z

gateway-messages/src/ereport.rs

+    Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, SerializedSize,
+)]
+#[repr(transparent)]
+pub struct RestartId(pub u128);


I did consider using the uuid crate for this, as it supports no_std, but it didn't really seem worth adding another dependency that would have to be compiled in to the Hubris binaries basically just to get UUID-like formatting in Debug impls that are only used by MGS...

hawkw · 2025-04-07T17:20:28Z

gateway-messages/src/ereport.rs

I felt that it was nicer to put all the ereport messages (both the SP-to-MGS messages and the MGS-to-SP messages) in their own module, rather than putting some of them in sp_to_mgs and others in mgs_to_sp. This way, a reader interested in the ereport stuff need only read this module, and a reader interested in the control-plane-agent protocol doesn't have to scroll past ereport messages. Future additions to the ereport protocol would change only the code in this file.

hawkw · 2025-04-07T17:21:41Z

gateway-sp-comms/src/ereport.rs

Similarly to https://github.com/oxidecomputer/management-gateway-service/pull/370/files#r2031615326, it felt nicer to have all the ereport bits defined in their own module, rather than smeared across shared_socket.rs and single_sp.rs. That way, all the ereport-specific code is in one place and it's easier to see the relationship between the code in the ereport socket receive handler and the single-SP handler, rather than having to trace it between modules.

No argument from me. Do you think it would be clearer to move the control-plane-agent stuff to its own submodule too? (Not as part of this PR, but maybe alongside the renamings I suggested in another comment?)

I'm definitely open to doing that in a subsequent change, I think it seems pretty reasonable (especially if we ever add a third socket/protocol for some other thing). I agree that we shouldn't mess with the control-plane-agent stuff in this PR though.

hawkw · 2025-04-07T17:23:57Z

gateway-sp-comms/Cargo.toml

@@ -27,7 +30,7 @@ thiserror.workspace = true
 tlvc.workspace = true
 tokio.workspace = true
 usdt.workspace = true
-uuid.workspace = true
+uuid = { workspace = true, features = ["v4"] }


This was necessary for unrelated reasons: the task dump code added in #316 uses Uuid::new_v4 in gateway-sp-comms, but gateway-sp-comms doesn't enable the "v4" feature (only faux-mgs does). So, this didn't compile for me using the v2 cargo feature resolver.

hawkw · 2025-04-07T17:45:32Z

gateway-sp-comms/src/ereport.rs

+                    Some(CborValue::Integer(i)) => task_names
+                        .get(i as usize)
+                        .cloned()
+                        .ok_or(DecodeError::BadTaskNameIndex {
+                            n,
+                            index: i as usize,
+                        })?,


@cbiffle this is what I was asking about in https://github.com/oxidecomputer/rfd/pull/849#discussion_r2027376877: right now, this code assumes that when an ereport's task name is an integer, it will always be the index of an ereport that was earlier in the packet than the current one. I wanted to get your confirmation of whether we could rely on that assumption or would need to handle indexes pointing ahead of the current ereport.

@cbiffle

per yesterday's chat with @cbiffle

i figured out how to force Serde to always use the "indefinite" encoding, so we can have some more tests

jgallagher

I didn't look particularly closely at the ereport parsing etc details (I'll defer that to you and Cliff, if that's okay). The structural MGS changes look good; just a handful of nits and questions.

jgallagher · 2025-04-10T17:28:09Z

gateway-sp-comms/src/shared_socket.rs

    recv_handler_task: JoinHandle<()>,
    log: Logger,
 }

-impl Drop for SharedSocket {
+// Hand-rolled `Debug` impl as the message type (`T`) needn't be `Debug` for the
+// `SharedSocket` to be debug.


This seems fine, but I'll plug https://docs.rs/derive-where/latest/derive_where/ since we use it in omicron, if you want to pull it in here too.

jgallagher · 2025-04-10T17:34:49Z

gateway-sp-comms/src/single_sp.rs

@@ -282,15 +284,18 @@ details in `dump.json`."
 pub struct SingleSp {
    interface: String,
    cmds_tx: mpsc::Sender<InnerCommand>,
+    ereport_req_tx: mpsc::Sender<ereport::WorkerRequest>,


I would not do this as a part of this PR, but I'm curious for your thoughts: there are a bunch of places like this one where we're going from one "thing" (in this case, an mpsc::Sender) to two "things". The original "one" is always named generically, and the new one is named indicating it's related to ereports. Do you think we should go back and rename the generic things to indicate they're intended for control-plane-agent?

Yeah, I think it might be worthwhile to do that (especially if we ever add a third port to the management network...). You're right that I just didn't really want to touch all the control-plane-agent code in this PR, but I'd be happy to do it in a separate commit.

jgallagher · 2025-04-10T17:36:43Z

gateway-sp-comms/src/ereport.rs

No argument from me. Do you think it would be clearer to move the control-plane-agent stuff to its own submodule too? (Not as part of this PR, but maybe alongside the renamings I suggested in another comment?)

gateway-sp-comms/src/ereport.rs

faux-mgs/src/main.rs

i thought i had gotten rid of these...

@jgallagher

as per [this comment][1] from @jgallagher. this is similar to the control-plane-agent protocol. wow, it's almost like we're reimplementing TCP (but without flow control because thats hard). [1]: #370 (comment)

hawkw added 2 commits March 27, 2025 10:03

mgs_to_sp: add EreportRequest message

e7b8fdc

sp_to_mgs: ereport resposne headers

1c1e7fa

hawkw requested a review from cbiffle March 27, 2025 23:08

hawkw added 23 commits March 27, 2025 16:25

ORDER MATTERS

41cfbdd

s/ReporterGeneration/Restart

75a845c

allow ereport response packets to provide metadata

8dee8f1

make the ascii diagram either better or worse

b05fca9

this one missed the big renaming

121640c

recv message plumbing

9719ace

reticulating SingleSp

1adbcd1

huge pile of parsing stuff

36350fe

add SingleSp::ereports

ab91de0

quick test for ereport decoding, fix some bugs

8869065

slightly nicer decode errors

c7e9e3b

fixup faux-mgs

871a785

reduce annoying generics

d4de62c

need this part

fab7101

nicer error

b13af93

don't keep re-requesting old restart ID

6b7c1bf

oopsie

make ereport errors a little nicer for MGS

df8f30e

move all ereport messages into ereport module

df96c18

de-stutter names

79ee2a9

ereport types naming tweaks

fbea81d

docs embetterment

da36a5d

hex format enas like the control plane does

663df64

hawkw changed the title ~~Ereport protocol messages~~ Ingest ereports from SPs Apr 6, 2025

hawkw added 3 commits April 6, 2025 10:06

factor out shared socket RecvHandler recv bits

9150029

reorganize and document shared socket recv bits

79a344e

docs improvement

b0c55c8

hawkw added 2 commits April 7, 2025 09:26

add tests that enum serialized reprs don't change

7535bcf

add module level docs

74ec442

hawkw commented Apr 7, 2025

View reviewed changes

handle cbor bytes

99b8b40

hawkw commented Apr 7, 2025

View reviewed changes

hawkw requested review from mkeeter and jgallagher and removed request for mkeeter April 7, 2025 17:41

hawkw marked this pull request as ready for review April 7, 2025 17:41

hawkw commented Apr 7, 2025

View reviewed changes

hawkw added 6 commits April 7, 2025 15:31

add limit field to requests

1522c6a

force use of checked request methods

58d073a

make CBOR-to-JSON conversion do what the RFC says

bc687b9

new response encoding

d39405f

per yesterday's chat with @cbiffle

make ascii diagrams sort of goof

83d252c

more tests

e74ed21

i figured out how to force Serde to always use the "indefinite" encoding, so we can have some more tests

jgallagher reviewed Apr 10, 2025

View reviewed changes

hawkw added 4 commits April 10, 2025 11:29

review feedback from @jgallagher

485d404

delete duplicate old ereport types

dec98f3

i thought i had gotten rid of these...

add request/response sequence numbers

8a49211

as per [this comment][1] from @jgallagher. this is similar to the control-plane-agent protocol. wow, it's almost like we're reimplementing TCP (but without flow control because thats hard). [1]: #370 (comment)

add some more commentary

c26dad3

hawkw requested a review from jgallagher April 10, 2025 20:23

fix MGS always returning the requested restart ID

41f0017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ingest ereports from SPs #370

Ingest ereports from SPs #370

hawkw commented Mar 27, 2025 •

edited

Loading

hawkw Apr 7, 2025

hawkw Apr 7, 2025

hawkw Apr 7, 2025

jgallagher Apr 10, 2025

hawkw Apr 10, 2025

hawkw Apr 7, 2025

hawkw Apr 7, 2025

jgallagher left a comment

jgallagher Apr 10, 2025

jgallagher Apr 10, 2025

hawkw Apr 10, 2025

jgallagher Apr 10, 2025

Ingest ereports from SPs #370

Are you sure you want to change the base?

Ingest ereports from SPs #370

Conversation

hawkw commented Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgallagher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hawkw commented Mar 27, 2025 •

edited

Loading