Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add kafka connection properties to profile #791

Merged
merged 4 commits into from
Dec 3, 2024

Conversation

emef
Copy link
Contributor

@emef emef commented Nov 22, 2024

this is to support adding additional rdkafka config params when creating a kafka connection profile

@@ -106,6 +106,16 @@
"sensitive": ["apiSecret"]
}
]
},
"connectionProperties": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a bit better as as map instead of a vec so that users don't need to know the k/v syntax. This is how we do property overrides on the table:

"client_configs": {
"type": "object",
"title": "Client Configs",
"description": "Additional Kafka configs to pass to the underlying Kafka consumer or producer. See [Kafka Consumer Configs](https://kafka.apache.org/documentation/#consumerconfigs) and [Kafka Producer Configs](https://kafka.apache.org/documentation/#producerconfigs) for more details.",
"additionalProperties": {
"type": "string"
}
},

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 agreed map is better

bail!("invalid connection property: {}", prop);
}
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change we'll be able to set client configs from both the connection and the table. I think it would be good to warn when iterating through the table configs (on 939) if both define the same property.

Copy link
Member

@mwylde mwylde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@mwylde mwylde merged commit 868b3d5 into ArroyoSystems:master Dec 3, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants