You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The background for this is described in estuary/flow#192. The goal is to be able to verify whether a given consumer.Checkpoint is from a particular Gazette cluster, so that we could identify when the offsets in the checkpoint would be nonsensical.
The thought is for Gazette to generate a unique id and put it at some pre-defined key relative to the --etcd.prefix (something like <etcd.prefix>/clusterID). Each member can do this in a transaction, and just let etcd decide which member wins the race. The cluster id should only ever be stored in etcd once, and never updated after that. The value of the cluster id should then be included in all Checkpoints.
An open question is where and when to verify that the cluster id in etcd matches the cluster id provided as part of a checkpoint. One possibility would be for the consumer framework to validate the return value of RestoreCheckpoint by checking its cluster id. Regardless, we need to ensure that a checkpoint with a missing cluster id is always accepted, so we don't break compatibility with previous versions.
The text was updated successfully, but these errors were encountered:
The background for this is described in estuary/flow#192. The goal is to be able to verify whether a given
consumer.Checkpoint
is from a particular Gazette cluster, so that we could identify when the offsets in the checkpoint would be nonsensical.The thought is for Gazette to generate a unique id and put it at some pre-defined key relative to the
--etcd.prefix
(something like<etcd.prefix>/clusterID
). Each member can do this in a transaction, and just let etcd decide which member wins the race. The cluster id should only ever be stored in etcd once, and never updated after that. The value of the cluster id should then be included in allCheckpoint
s.An open question is where and when to verify that the cluster id in etcd matches the cluster id provided as part of a checkpoint. One possibility would be for the consumer framework to validate the return value of
RestoreCheckpoint
by checking its cluster id. Regardless, we need to ensure that a checkpoint with a missing cluster id is always accepted, so we don't break compatibility with previous versions.The text was updated successfully, but these errors were encountered: