-
Notifications
You must be signed in to change notification settings - Fork 229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add kafka connection properties to profile #791
Conversation
@@ -106,6 +106,16 @@ | |||
"sensitive": ["apiSecret"] | |||
} | |||
] | |||
}, | |||
"connectionProperties": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would be a bit better as as map instead of a vec so that users don't need to know the k/v syntax. This is how we do property overrides on the table:
arroyo/crates/arroyo-connectors/src/kafka/table.json
Lines 83 to 90 in a9ac8f7
"client_configs": { | |
"type": "object", | |
"title": "Client Configs", | |
"description": "Additional Kafka configs to pass to the underlying Kafka consumer or producer. See [Kafka Consumer Configs](https://kafka.apache.org/documentation/#consumerconfigs) and [Kafka Producer Configs](https://kafka.apache.org/documentation/#producerconfigs) for more details.", | |
"additionalProperties": { | |
"type": "string" | |
} | |
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 agreed map is better
bail!("invalid connection property: {}", prop); | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With this change we'll be able to set client configs from both the connection and the table. I think it would be good to warn when iterating through the table configs (on 939) if both define the same property.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great!
this is to support adding additional rdkafka config params when creating a kafka connection profile