various updates to capture & materialization protocols #5
Conversation
Re-define driver checkpoints to be patches which are applied to a fully reduced driver checkpoint managed by the Flow runtime. Issue #173
RecvMsg is more efficient, especially as connectors are factored into images. Issue #173
Allow users of the `flow` package to directly access Gazette types rather than also having to import those Gazette protocol packages.
jgraettinger
requested review from
a team and
snowzach
and removed request for
a team
There was a problem hiding this comment.
Everything seems pretty reasonable. I was honestly expecting there to be more changes, but I think that was my misunderstanding about how far along we were with the Materialization protocol. Reviewing this helped put a lot of specifics to the concepts I've heard us discuss. 👍
| json_name = "driverCheckpoint" | ||
| json_name = "driverCheckpointMergePatch" |
There was a problem hiding this comment.
The idea here is that a full checkpoint will "patch" into any previous state with by overwriting everything, which while redundant, should still be correct, right?
Is there ever a circumstance that we will need to differentiate actual patches from full state checkpoints?
There was a problem hiding this comment.
The idea here is that a full checkpoint will "patch" into any previous state with by overwriting everything, which while redundant, should still be correct, right?
That's right.
Is there ever a circumstance that we will need to differentiate actual patches from full state checkpoints?
- It means a checkpoint can no longer represent object properties which are intentionally null.
- It also means that intentional deletion of fields by omitting them in a future checkpoint no longer works.
That said, I'm not aware of any current connectors where these are a problem, and the Airbyte folks say they're on board. So likely to change in the future but 🤷
| // - The driver server sends TransactionResponse.Opened after ensuring | ||
| // that other raced clients are fenced (if fencing is supported). |
There was a problem hiding this comment.
I think this is probably a bit tangential, but what exactly do you mean by "fencing" in this context? I'm not familiar with this terminology.
There was a problem hiding this comment.
Terminology comes from memory barriers. A fence is an operation which inserts a barrier such that a (distributed) actor F is assured no other actor Z (often called a "zombie") is able to race a modification of a mutually shared resource. F has "fenced" the resource from Z.
| if count != 1 { | ||
| return pb.NewValidationError("expected one of Open, Load, Prepare, Store, Commit") | ||
| } |
There was a problem hiding this comment.
Is this a common go idiom for treating a struct like an enum?
There was a problem hiding this comment.
I'm not sure I've seen enough of it to call it idiomatic. Mostly I don't see it checked, and it makes me
Generics are finally coming to Go, but I'd dearly love me some Rust-like algebraic enums and pattern matching....
Related to factoring out materialization connector images.