Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support automatic table schema evolution #277

Open
mlivirov opened this issue Dec 6, 2023 · 2 comments
Open

Support automatic table schema evolution #277

mlivirov opened this issue Dec 6, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@mlivirov
Copy link
Contributor

mlivirov commented Dec 6, 2023

Description

In my organization we use clickhouse as analitical database that gets it's data from multiple sources via Kafka. Domain models change from time to time as business changes and we found a need to reflect those changes in the clickhouse manually a little inconvenient.

Would be awesome if this connector could propagate changes in schemas to the clickhouse tables automatically.

Additional context
Compatibility modes in kafka:
https://docs.confluent.io/cloud/current/sr/fundamentals/schema-evolution.html#compatibility-types

JDBC connector that has this functionality:
https://github.com/confluentinc/kafka-connect-jdbc

@mlivirov mlivirov added the enhancement New feature or request label Dec 6, 2023
@Paultagoras
Copy link
Contributor

Hi @mlivirov ! So this seems like it would have a race condition, if there are multiple tasks running - is that not the case with the jdbc connector mentioned?

@mlivirov
Copy link
Contributor Author

mlivirov commented Jun 5, 2024

Not sure how it's done in jdbc connector. Wouldn't it be possible to use separate KeeperMap table to synchronize tasks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants