Independently defined events

[matejpavlovic]

Creating this discussion from @xosmig 's original question:

Currently, there is no way to define Mir event types outside Mir (i.e., Mir cannot be used as a library, only as a fork). Any ideas on how we may eventually solve this? I am raising this issue now because it may make sense to take it into account in the code generator.

[matejpavlovic]

This is a good point and will need to be addressed. The core of this problem seems to be that the Mir Event protobuf type uses a oneof that includes all the subtypes of Event. As long as the definition of the root Event type is static and resides in the Mir code base, it seems impossible to add new subtypes of Event without modifying that definition.

The first approach that comes to my mind is to generate the .proto file that contains the definition of Event (currently protos/eventpb/eventpb.proto). Mir as a library would contain a little tool that could be told where the external event type definitions are, identify all the subtypes of Mir events, generate the definition of Event in a .proto file and only then compile that .proto.

[xosmig]

This approach would still not be exactly the same as simply including the Mir library in the go.mod file

I think requiring to run a code generation tool is perfectly fine.

[matejpavlovic]

The same way as now. The only difference would be that if we want to create a new event type now, we need to go to the eventpb.go file and manually add it like this

message Event {
  oneof type {
    option (mir.event_type) = true;

    Init                    init                    = 1;
    Tick                    tick                    = 2;
    WALAppend               wal_append              = 3;
    ...                     ...                     = ...;
    NewEventType            new_event_type          = zyz;

I thought about automating this by creating a tool that generates this file, generates the protobufs, and compiles Mir with the newly generated protobuf files.

[xosmig]

The same way as now

Now the definition of eventpb.Event exists within Mir repo. If I understand your proposal correctly, the users of the Mir library will generate their own eventpb.Event, but how would they tell Mir which eventpb.Event to use without modifying the Mir code?

[xosmig]

It's like we want Mir, as a library, to be parameterized by the event type.

[matejpavlovic]

Well, what I meant is that the code generation tool would know the location of the Mir library and actually replace the local-path-to-library/protos/eventpb/eventpb.proto file there. I know that this is still sub-optimal, it's just a small step further from actually having to go and manually modify Mir code for adding events.

[matejpavlovic]

Another option would be define a separate abstraction for events, probably completely independent of protobufs, that would take care of all the serialization / deserialization... Not sure that's the best way to go, but it might be worth thinking about. If we choose to go this way, there would be some significant refactoring overhead.

[xosmig]

If we choose to go this way, there would be some significant refactoring overhead

This is something I thought maybe the code generation tool I am currently working on could help us with. It could convert the current format to a new format. Not sure what the new format would be though.

We could also make this abstraction in such a way that the current implementation of events satisfies it (maybe, with methods Serialize/Deserialize added to *eventpb.Event or to each event type (with some help from the code generator)).

[xosmig]

One important aspect to take into account: it should be possible to make mir modules as libraries as well.

[matejpavlovic]

This approach is definitely the cleaner one of the two discussed so far, we'd just need to figure out a proper basic format, especially the format of encoding the event type (to replace the static protobuf definition).
An easy way could, for example, to have explicit names for event types encoded as strings (possibly with some reservations for Mir system events) and every custom event would need to be able to serialize itself to some encoding of {name: string, data: byte array}, possibly even directly to a protobuf message

message CustomEvent {
  string type = 1;
  bytes  data = 2;
}

Or, generalizing even further, we could just define Mir's Event type like this

message Event {
  bytes          data        = 1;
  string         dest_module = 2;
  repeated Event next        = 3;
}

In this case, each module would only be responsible to produce and consume these top-level events and be completely free to implement any serialization format it wants. If modules wanted to interoperate, they would need to make sure to understand each other's events. If this option is too extreme and could create anarchy between different module's event formats, A middleground could go like this

message Event {
  ModuleEvent    data        = 1;
  string         dest_module = 2;
  repeated Event next        = 3;
}

message ModuleEvent {
  // Some fields every module event needs to use.
}