Projecting a subset of events

[flyingpie]

Hey guys, thank you for the work on this excellent library!

We've set up a system where event types are always the same ("MyEvent" in these examples), so that events can be published without any changes to event ingestion.

This allows us to start publishing events before the projections are in place, and without needing to know any data types around the newly created events (data is passed along as JObject properties, which get parsed during projection).

Each event contains a "stream type", which is what we'd like to use to distinguish events on. I assume that this "stream type" is analogous to what you'd normally use the CLR event type for, and/or the CLR type when calling StartStream<>().

So on the projection side, I'm trying to have a projection only apply to a subset of events, using this "stream type", instead of event CLR type or StartStream<> type.

So far, these attempts I've tried:

Simple SingleStreamProjection, which works, but projects all events, regardless of the "stream type". At the moment Project is called, an aggregate object (will) already exist in the database.

public class MyAggregate : SingleStreamProjection<MyProjection>
{
  public MyAggregate()
  
    ProjectEvent<IEvent<MyEvent>>((querySession, projection, @event) =>
    {
      // Projection as per usual.
    });
  }
}

EventProjection, which allows us to prevent aggregate docs from being stored, and all in all works quite well.

public class MyAggregate : EventProjection
{
  public MyAggregate()
  {
    ProjectAsync<IEvent<MyEvent>>(
      async (ev, ops) =>
      {
        if (!ev.Data.StreamType.Equals("some stream type", StringComparison.OrdinalIgnoreCase))
        {
          return;
        }

        var doc = await ops.LoadAsync<MyProjection>(ev.Data.StreamId);

        doc ??= new MyProjection()
        {
          Id = ev.Data.StreamId,
          TenantId = ev.Data.Tenant,
          ...
        };

        ops.Store(doc);
      });
  }

}

It's one thing that unrelated events are still processed (would be nice to cap that off at the database level of course). But more problematic for now, is that on projection rebuilds, the existing database objects are not removed.

I've started looking into custom projections, and possibly assigning a different CLR event type on runtime somehow, but I have a feeling that there should be a pretty simple way of doing this, hence this question.

Any help is appreciated!

[flyingpie]

Additionally, here's the MyEvent definition:

public class MyEvent
{
	[Identity]
	public Guid Id { get; set; } = Guid.NewGuid();

	public string? StreamType { get; set; } // <-- We'd like to apply projections on events with a specific value in here.

	public string? StreamId { get; set; }

	public string? EventId { get; set; }

	public string? EventType { get; set; }

	public JObject? EventData { get; set; }

	public string? Tenant { get; set; }

	public DateTimeOffset Timestamp { get; set; } = DateTimeOffset.UtcNow;
}

[jeremydmiller]

I moved this to a discussion as there's nothing actionable from the Marten codebase side here

[jeremydmiller]

@flyingpie Sorry, this just isn't a supported -- or recommended -- way to use Marten events or projections. You could maybe try setting the StreamType property of your projections to make the projection rebuilds filter on the stream type in the database (the T in StartStream()). For Marten usage, you would have been better off using semantically different event types.

Why did you go down this path btw? Was this a conversion from some previous event store?

And not to be a jerk about this, but you have to understand that we cannot support this usage that so far outside the mainstream of how Marten is meant to be used.

[flyingpie]

@jeremydmiller Thank you for your comment!

I can understand your point of view, that it goes against the heavily-typed patterns in Marten, no offense taken.

You could say that it's due to a previous event store, where everything is not as statically typed.
The dynamic side of the system is publishing the events, which we then want to project typed.

We mainly want to use Marten for the projection part I guess, for which it's beautifully serving its purpose.

Thank you for your time, Jeremy, useful to know what is and isn't part of the scope. Looking forward for what comes next!