Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a cluster id to consumer Checkpoint #292

Open
psFried opened this issue Aug 18, 2021 · 0 comments
Open

Add a cluster id to consumer Checkpoint #292

psFried opened this issue Aug 18, 2021 · 0 comments

Comments

@psFried
Copy link
Contributor

psFried commented Aug 18, 2021

The background for this is described in estuary/flow#192. The goal is to be able to verify whether a given consumer.Checkpoint is from a particular Gazette cluster, so that we could identify when the offsets in the checkpoint would be nonsensical.

The thought is for Gazette to generate a unique id and put it at some pre-defined key relative to the --etcd.prefix (something like <etcd.prefix>/clusterID). Each member can do this in a transaction, and just let etcd decide which member wins the race. The cluster id should only ever be stored in etcd once, and never updated after that. The value of the cluster id should then be included in all Checkpoints.

An open question is where and when to verify that the cluster id in etcd matches the cluster id provided as part of a checkpoint. One possibility would be for the consumer framework to validate the return value of RestoreCheckpoint by checking its cluster id. Regardless, we need to ensure that a checkpoint with a missing cluster id is always accepted, so we don't break compatibility with previous versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant