Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Topic creation improvements #70

Open
1 of 2 tasks
markpapadakis opened this issue Oct 9, 2019 · 1 comment
Open
1 of 2 tasks

Topic creation improvements #70

markpapadakis opened this issue Oct 9, 2019 · 1 comment
Assignees
Labels

Comments

@markpapadakis
Copy link
Member

markpapadakis commented Oct 9, 2019

Now, when not running in cluster-aware mode, the directory name for the topic will be prepended with '.', and only when all partitions for that topic have been initialized (in that .directory), that .directory will be atomically renamed to the correct name(i.e the topic name).

This is important because it's possible that partition creation may fail(e.g system fault, FS fault, etc), and we don't want to leave a mess when that happens. That is to say, creating a topic should be an atomic operation.

  • implement cleanup() lambda in Service::process_create_topic()
  • implement the same idea in Service::reconcile_cluster_topology()
@markpapadakis markpapadakis self-assigned this Oct 9, 2019
@markpapadakis
Copy link
Member Author

Turns out that deleting directories on startup may lead to data loss - because TANK may be misconfigured to use the wrong base directory. Will now generate a warning and will eventually accept a new command-line argument enabling GC on startup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant