Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python SDK: Support multiple pydantic versions #7672

Merged
merged 7 commits into from
Apr 15, 2024

Conversation

N-o-Z
Copy link
Member

@N-o-Z N-o-Z commented Apr 14, 2024

Closes #6903

Change Description

Background

Support pydantic v1/2 on python SDK and HL SDK.

New Feature

We use templates to untag the pydantic version together with a post processing script which changes all imports to try and use the pydantic.v1 compatible module (in pydantic v2) and fallback on pydantic v1 modules

Testing Details

Tested on HL SDK integration tests with pydantic v2 installed

Breaking Change?

No

@N-o-Z N-o-Z self-assigned this Apr 14, 2024
@N-o-Z N-o-Z added the include-changelog PR description should be included in next release changelog label Apr 14, 2024
Copy link

E2E Test Results - DynamoDB Local - Local Block Adapter

13 passed

Comment on lines 6 to 14
echo "... $line ..."
modules=$(echo $line | awk -F 'from pydantic import ' '{print $2}')
echo $line

newline="try:\n from pydantic.v1 import $modules\nexcept ImportError:\n from pydantic import $modules"

echo "$newline"

echo "Replacing import statement"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think all the echos are great for debugging but kind of redundant, maybe just output the file

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not printed during the build process. It is used when running the script manually for testing

Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat! Much value in few lines

  1. How was this tested?
  2. How can we test this on DataBricks?

Co-authored-by: Ariel Shaqed (Scolnicov) <ariels@treeverse.io>
Copy link
Collaborator

@ozkatz ozkatz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THANK YOU!

@N-o-Z
Copy link
Member Author

N-o-Z commented Apr 15, 2024

Neat! Much value in few lines

  1. How was this tested?
  2. How can we test this on DataBricks?

So, I thought about doing it in a separate PR but since you asked. I created a matrix for the wrapper unit tests workflow.
One will run with pydantic v2 while the other with pydantic v1 (exact version taken from the AWS DBR 15 dependency - BTW in Azure DBR 12.2 there's no pydantic dep.)
However this will take affect only after we release a new SDK client and modify the HL SDK dep.

@N-o-Z N-o-Z merged commit b966e75 into master Apr 15, 2024
36 checks passed
@N-o-Z N-o-Z deleted the task/support-multiple-pydantic-versions-6903 branch April 15, 2024 14:41
emulatorchen pushed a commit to emulatorchen/lakeFS that referenced this pull request May 27, 2024
* Python SDK: Support multiple pydantic versions

* Fix templates

* Update clients/python/scripts/pydantic.sh

Co-authored-by: Ariel Shaqed (Scolnicov) <ariels@treeverse.io>

* Add unit test flavor with pydantic V1

* Fix script

* Fix workflow

---------

Co-authored-by: Ariel Shaqed (Scolnicov) <ariels@treeverse.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/sdk/python include-changelog PR description should be included in next release changelog python-sdk-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python SDK (lakefs_sdk) depends on outdated pydantic 1.X and prevents migration from lakefs_client
4 participants