'split brain' event sourcing

[danieleades]

first of all, thankyou @ar3s3ru for your infinite patience...

I have a use case i wanted to run by you and get your thoughts on. both generically and with respect to this library.

I have a system made up of multiple 'nodes' connected only sporadically, and with very limited bandwidth. Some of the event streams of each node are entirely independent, while others represent a shared state, or world model shared across the nodes. For a shared event stream, every now and again the nodes will have an opportunity to synchronise events between themselves. This is a so-called 'split brain' model, where the nodes have a single, shared 'brain', but it must tolerate being separated for potentially large proportions of the time. To illustrate-

1. node A produces events: [A1, A2, A3]
2. node B simultaneously produces events: [B1, B2, B3]
3. at some point after that, they are able to synchronise, such that each node now has the same event stream (interleaved by timestamp): [A1, B1, A2, B2, A3, B3]
4. the aggregates MUST be able to deconflict the messages infallibly such that after synchronisation node A and node B agree on the current state.

this would be particularly well-suited to very simple cases like 'position of an asset', since state update strategy is just to consider the most recent event. But it could also work for more complex situations.

do you think event sourcing makes sense as a model for this kind of eventual consistency? the closest equivalent i'm aware of is CouchDB's multi-master replication. Rather than event sourcing, it uses a slightly more complex model which tracks a tree of revisions to the original document (though you could replicate this using event sourcing with a couple of extra uuids to track and a slightly clever aggregate).

I could imagine this might be challenging with this library, because we have to drop the assumption that the events have monotonically increasing sequence IDs. also, the Projections would need a mechanism for invalidating their cached state when the history of events is changed by this synchronisation.

[ar3s3ru]

Sounds to me like both node A and node B are using the same Aggregate. If so, changes to an Aggregate should already be ordered (due to optimistic concurrency used with the version).

In this case, to implement conflict resolution you can simply implement some retry mechanism at the Command Handler level, so that if a Command fails due to a conflict error, you just submit the same Command again and it will be evaluated using the new state of the Aggregate.

Example:

1. Command(A1) and Command(B1) in-flight,
2. Command(A1) completes execution first, writes to AggregateEventStream
3. AggregateEventStream is now: [A1]
4. Command(B1) tries to write to AggregateEventStream as [B1], but the state is [A1] so it fails.
5. Conflict error is detected, Command(B1) gets retried again by the Command Handler
6. Command(B1) completes execution now
7. AggregateEventStream is now: [A1, B1]

---

Another way is to use separate Aggregates for Node A (let's call it AggregateA) and Node B (AggregateB), but to use a Projection/Process Manager to write to an additional Aggregate (AggregateC).

Event Sourcing adopts stream processing similar to other approaches/technology (e.g. Event Streaming with Kafka); it's important that, when streaming events, the ones for AggregateA and AggregateB end up "repartitioned" to the same shard/partition, so that you can consume them in order.

Once you have that, it's all about performing a left-fold of those events into the additional Aggregate.

---

You can implement this with this library, might be easier however to just adopt the first option: same aggregate. For the second option, we need to rework this library first. Ideally you could be able to do so with 0.5.0 😄

Not sure if this was clear enough, it's hard to explain all of this without having a tangible example to use (not just Node A/B, events A/B) 😅

If not, I'd ask you to give me a more concrete example and I can try to rephrase the options here above 👍🏻

[danieleades]

Thanks for taking the time! I'll try to out together a more concrete example. But to frame the discussion-

• events are produced at a frequency of order seconds
• 'nodes' may be separated from each other for durations on the order of minutes or even hours

[ar3s3ru]

Got it. Do the nodes share a data store, or do they have local stores?

When they "reconcile" their state, where does the "unreconciled" state live? And where does the reconciled one?