Event log

[mistermoe]

First of 3 PRs to implement sync. Introduces an EventLog interface along with a concrete level-based implementation. Events are added to the EventLog for:

• RecordsWrite
• RecordsDelete
• ProtocolsConfigure

The PR that follows this one will introduce the EventsGet interface method

[diehuxx]

Codewise this looks generally good to me, with a couple small questions. But I have some broader design questions before I stamp.

Let me know if you'd prefer design discussion here or in your tech proposal.

Trying to think through this scenario.

1. Say that DWN A and B both have record R. Initially they have the same most recent update of record R.
2. Then DWN A receives a RecordsWrite(A) updating R to be R(A). Slightly after that, before any sync occurs, DWN B also receives a RecordsWrite(B) updating R to be R(B).
3. Is it deterministic which version of R wins? If not, is that acceptable?

If DWN A initiates a pull from B, then it will send the CID of RecordsWrite(A) to DWN B in an EventsGet. DWN B has never seen this message, so I’m actually not sure what it will return, since it’s supposed to return events after the watermark. Does it return no events? Or all events from all time?

[mistermoe]

@thehenrytsai i went ahead and included functional tests. pls lmk if you'd rather them go elsewhere.

[mistermoe]

just realized that we'll need to delete events when a protocol is overwritten as well. Specifically, here. 😭

[mistermoe]

just realized that we'll need to delete events when a protocol is overwritten as well. Specifically, here. 😭

done

[mistermoe]

does anyone have any reservations left here?

[thehenrytsai]

The only reservation for me is that the code does a full scan of the event log for every clobber, but fine if we create an issue to improve event deletion by CIDs as a separate item to unblock the merge of this PR.

[mistermoe]

@thehenrytsai i'm now storing events in watermark order and cid order. this prevents us from having to scan a tenant's entire event log when deleting by CID. cost incurred is 2x storage for events. the size of each event is numDidBytes + numCidBytes.

requests to change anything else about EventLogLevel can be submitted as separate issues that can be tackled later. I'd like to get this PR merged so that i can move to the next sync-related PR.

[csuwildcat]

Sounds like a good plan to me, Moe

[thehenrytsai]

🥇 🙏 💯 🎸