-
Notifications
You must be signed in to change notification settings - Fork 435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Capture Kafka key without using it as ID #704
Comments
Ideally, all Kafka message metadata should be captured in a way that is searchable: topic, partition, offset, timestamp, headers |
A year later and there is still no way to capture/search the most basic message metadata within ES. The only reason I wanted Kafka messages copied to ES was to search it. IMO this library is pretty much worthless. Kafka message keys are used for partitioning and are not intended to be unique. The fact that the implementors of this connector thought they should be used as an Elasticsearch document ID shows a pretty poor understanding of both domains. |
I want to use the topic/partition/offset as the ES document ID while still capturing the Kafka message key for audit/search purposes. Unless I'm mistaken, there doesn't appear to be a way to do this with this connector.
The text was updated successfully, but these errors were encountered: