fix: tweak datahub-kafka requirement #110
Conversation
This should enable pydantic 2 support Slack ref: https://datahubspace.slack.com/archives/CV2KB471C/p1706298872346799?thread_ts=1706201071.771369&cid=CV2KB471C
| @@ -33,7 +33,7 @@ def get_long_description(): | |||
| acryl_datahub_min_version = os.environ.get("ACRYL_DATAHUB_MIN_VERSION") or "0.12.1.2" | |||
|
|
|||
| base_requirements = { | |||
| f"acryl-datahub[kafka]>={acryl_datahub_min_version}", | |||
| f"acryl-datahub[datahub-kafka]>={acryl_datahub_min_version}", | |||
There was a problem hiding this comment.
It seems kafka and datahub-kafka both used kafka_common from metadata-ingestion/setup.py, how would this change support pydantic 2? I looked at the gradle build log for this PR and it looks like it first pulls pydantic pydantic-2.5.3 and then uninstalls it and installs pydantic-1.10.14 instead.
There was a problem hiding this comment.
All plugins except for a couple currently get a pydantic v1 pin https://github.com/datahub-project/datahub/blob/1498c36875450b1a1f44d53e8e8c47c41a91dc69/metadata-ingestion/setup.py#L737
I believe it installs pydantic v1 in the gradle build because it includes the snowflake_tag_propagation action that needs the acryl-datahub snowflake plugin, which needs pydantic v1
However, this should allow other usages of actions to not need pydantic v1
|
I think this is probably due to the update for lint, without changing those source codes, the build is failing with the following files with lint errors, rerun the `./gradlew lintFix will solve the issue: |
This should enable pydantic 2 support. The acryl-datahub kafka source still requires pydantic v2, but the datahub-kafka sink does not. We only need the sink capabilities here.
Slack ref: https://datahubspace.slack.com/archives/CV2KB471C/p1706298872346799?thread_ts=1706201071.771369&cid=CV2KB471C