Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Python 3.11 #60

Merged
merged 11 commits into from
Nov 28, 2022
Merged

Support Python 3.11 #60

merged 11 commits into from
Nov 28, 2022

Conversation

susodapop
Copy link
Contributor

@susodapop susodapop commented Oct 26, 2022 •
edited
Loading

Description

Status: This PR is on-hold awaiting PyArrow to release their pre-built wheels for Python 3.11.

This PR adds Python 3.11 to the unit testing matrix.

Closes #59

@susodapop
Copy link
Contributor Author

susodapop commented Oct 26, 2022 •
edited
Loading

The Python 3.11 tests fail because our numpy dependency is pinned to 1.21.1 but Python 3.11 support was added in 1.23.2

I'm going to split the dependency specification so that 1.21.1 is installed for Python 3.7 and a newer version is used for newer Pythons. This dependency resolution takes a long time on my workstation though. Will update this branch once the resolution completes.

@susodapop
Copy link
Contributor Author

Alright, I fixed the issue with numpy but now pyarrow is raising concerns 😬 . Will continue debugging and update this PR as I have more info.

@susodapop
Copy link
Contributor Author

Looks like the Pyarrow team are at work on this here apache/arrow#14499

@susodapop
Copy link
Contributor Author

I tried manually installing pyarrow in Python 3.11 on my Intel Mac and it worked. Only some of their wheels aren't building, including whatever GH actions needs to run its tests.

Jesse Whitehouse added 4 commits November 8, 2022 11:58
Add Python 3.11 to the unit testing matrix
f3605a5 
Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Updated poetry lock file to split our numpy dependency based on pytho…
3012d7e 
…n version

I installed the latest poetry (v1.2.2), deleted my lockfile, and ran
poetry update to generate these changes. I manually edited pyproject.toml
following the guidance from poetry's docs:

https: //python-poetry.org/docs/dependency-specification/#multiple-constraints-dependencies
Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Fix: unit tests ran twice for every push to a pull request
b779a30 
This revision runs tests once for a pull request and once for any push to
the main branch.

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Update lockfile after rebasing on main, including bumped thrift dep…
e6ec156 
…endency

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
@susodapop
Copy link
Contributor Author

I forced a push after rebasing on main which incorporates the new thrift dependency. I still expect the Python 3.11 build to fail because Arrow have not published wheels for Python v11 yet.

alexmalins
alexmalins reviewed Nov 25, 2022
View reviewed changes
pyproject.toml Outdated
@@ -14,8 +14,12 @@ thrift = "^0.16.0"
pandas = "^1.3.0"
pyarrow = "^9.0.0"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python 3.11 compatible wheels are for pyarrow versions >=10.0.1, so this line needs to change to make this PR work.

You could try using pyarrow = ">=9.0.0" or maybe better

pyarrow = [
    {version = ">=9.0.0", python = ">=3.7,<3.11"},
    {version = ">=10.0.1", python = ">=3.11"}
]

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, as @potiuk said in #59, the pyarrow wheels for Python 3.11 are now out.

I could install this package successfully from a local git repo clone by slacking off the above pyarrow requirement. Please see above comment

Jesse Whitehouse added 7 commits November 28, 2022 15:18
Add Python 3.11 to the unit testing matrix
ee1efcd 
Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Updated poetry lock file to split our numpy dependency based on pytho…
63cd0b6 
…n version

I installed the latest poetry (v1.2.2), deleted my lockfile, and ran
poetry update to generate these changes. I manually edited pyproject.toml
following the guidance from poetry's docs:

https: //python-poetry.org/docs/dependency-specification/#multiple-constraints-dependencies
Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Fix: unit tests ran twice for every push to a pull request
30dc4ff 
This revision runs tests once for a pull request and once for any push to
the main branch.

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Updated pyproject.toml to specify pyarrow dep by Python version
7d31aaf 
See #60 for discussion about why this was necessary

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Update poetry lockfile with dependency changes
e5b5cbd 
I installed the latest poetry (v1.2.2), deleted my lockfile, and ran
poetry update to generate these changes. I manually edited pyproject.toml
following the guidance from poetry's docs.

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Revert "Update lockfile after rebasing on main, including bumped th…
a870fd9 
…rift dependency"

This reverts commit e6ec156.

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
Merge branch 'python311-support-2' into python311-support
eac2b83 
Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
@susodapop
Copy link
Contributor Author

I rebuilt the lockfile with our latest change including @alexmalins revised pyarrow dependency spec. I'm watching the PR checks now to make sure they go through. Assuming they do, we'll bump our version and cut a new release in the next 24 hours.

@susodapop
Copy link
Contributor Author

Everything builds 🚀

I ran the e2e tests from Python 3.11 and Python 3.10.5 and all pass from PySQLCoreTestSuite.

@susodapop susodapop merged commit 26b9484 into main Nov 28, 2022
@susodapop susodapop deleted the python311-support branch November 28, 2022 21:47
@susodapop susodapop mentioned this pull request Nov 28, 2022
Merged
@potiuk
Copy link

potiuk commented Nov 28, 2022

Woooohooo ! I hope we get binary wheel in Pypi soon :)

@susodapop
Copy link
Contributor Author

Version 2.2.1 with Python 3.11 support is deploying to Pypi now: https://github.com/databricks/databricks-sql-python/actions/runs/3569089528

@susodapop
Copy link
Contributor Author

Update: v2.2.1 is now available on Pypi.

@potiuk
Copy link

potiuk commented Nov 28, 2022

Tomorrows news today :) . Thanks !

@susodapop
Copy link
Contributor Author

Thanks for your help and persistence on this!

@Tom-Newton Tom-Newton mentioned this pull request Dec 15, 2022
Closed
@ueshin ueshin mentioned this pull request Jan 11, 2023
Merged
ueshin added a commit to databricks/dbt-databricks that referenced this pull request Jan 12, 2023
@ueshin
Upgrade databricks-sql-connector to 2.2.2 (#250)
1dcd4e1 
### Description

Upgrades `databricks-sql-connector` to `2.2.2`.

It includes:

- Add tests for parameter sanitisation / escaping (databricks/databricks-sql-python#46) for the fix for #249
- Support Python 3.11 (databricks/databricks-sql-python#60)
ueshin added a commit to databricks/dbt-databricks that referenced this pull request Jan 12, 2023
@ueshin
Upgrade databricks-sql-connector to 2.2.2 (#250)
0e342aa 
### Description

Upgrades `databricks-sql-connector` to `2.2.2`.

It includes:

- Add tests for parameter sanitisation / escaping (databricks/databricks-sql-python#46) for the fix for #249
- Support Python 3.11 (databricks/databricks-sql-python#60)
ueshin added a commit to databricks/dbt-databricks that referenced this pull request Jan 12, 2023
@ueshin
Upgrade databricks-sql-connector to 2.2.2 (#250)
7e68e9a 
### Description

Upgrades `databricks-sql-connector` to `2.2.2`.

It includes:

- Add tests for parameter sanitisation / escaping (databricks/databricks-sql-python#46) for the fix for #249
- Support Python 3.11 (databricks/databricks-sql-python#60)
ueshin added a commit to databricks/dbt-databricks that referenced this pull request Jan 12, 2023
@ueshin
Upgrade databricks-sql-connector to 2.2.2 (#250)
43b426b 
### Description

Upgrades `databricks-sql-connector` to `2.2.2`.

It includes:

- Add tests for parameter sanitisation / escaping (databricks/databricks-sql-python#46) for the fix for #249
- Support Python 3.11 (databricks/databricks-sql-python#60)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexmalins alexmalins alexmalins left review comments

@moderakh moderakh moderakh approved these changes

@arikfr arikfr Awaiting requested review from arikfr

@yunbodeng-db yunbodeng-db Awaiting requested review from yunbodeng-db yunbodeng-db is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support for Python 3.11
4 participants
@susodapop @potiuk @moderakh @alexmalins