Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SQLAlchemy 2.0 support #510

Closed
toinbis opened this issue Nov 27, 2022 · 13 comments
Closed

SQLAlchemy 2.0 support #510

toinbis opened this issue Nov 27, 2022 · 13 comments
Assignees
@kiraksi
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@toinbis
Copy link

toinbis commented Nov 27, 2022

Hey there!

SQLAlchemy has already released a beta3 for 2.0, which would indicate the final 2.0 release is approaching quickly.

Currently sqlalchemy-bigquery support only <=1.4.27, if I am not mistaken. Am wonder is there a plan to support SQLAlchemy 2.0?

Thanks!
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label Nov 27, 2022
@tswast tswast added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Nov 29, 2022
@tswast
Copy link
Collaborator

tswast commented Nov 29, 2022

Thanks for the report! I think we're good now for 1.4.x versions, but I noticed in #513 that there are some failures we'll need to address for 2.x support. We'll keep this issue open while we figure out what's needed.

@tswast tswast mentioned this issue Nov 29, 2022
Closed
@Giuzzilla
Copy link

Giuzzilla commented Jan 7, 2023 •
edited
Loading

I think that a first step to go forward with this would be forcing the usage of SQLAlchemy >= 1.4 first (instead of >=1.2), this would allow using the new 2.0 syntax within the library (e.g. positional args in select statements vs iterables). https://docs.sqlalchemy.org/en/14/changelog/migration_20.html

@SlavaSkvortsov
Copy link

Hello!

Do you have any plans to unblock SQLAlchemy 2.0? It would be really helpful :)

Thank you!

@sar009
Copy link

sar009 commented Apr 13, 2023

Hi @tswast, SQLAlchemy 2.0 is out of beta now, can bigquery-sqlalchemy support 2.0

@meredithslota
Copy link

Hi folks — we don't have support for 2.0 planned yet, as there are some competing priorities internally. Apologies on the delays; we do not have an ETA yet for this.

@dominik-reiner dominik-reiner mentioned this issue Jun 29, 2023
Closed
@alexjironkin alexjironkin mentioned this issue Jun 30, 2023
Closed
@alexjironkin
Copy link

This is now a blocker for us to migrate to newer versions of both flask_sqlalchemy and sqlalchemy. When will you guys be able to look into supporting sqlalchemy v2? After all 1.4 is now a legacy version receiving minimal support.

@bpafoshizle
Copy link

Would be great to have this. This is preventing using langchain and llama-index for LLM development against big query as those require SQLAlchemy > 2.0. This means we may be pulling data out of BQ into e.g. postgres to build our LLM apps.

@richard-sparq
Copy link

@bpafoshizle 100% same here

@cofin cofin mentioned this issue Aug 26, 2023
Closed
@qrtt1 qrtt1 mentioned this issue Aug 28, 2023
Merged
2 tasks
@cofin cofin mentioned this issue Oct 15, 2023
Open
@sharoonthomas
Copy link
Contributor

@meredithslota @tswast I have a patch for supporting 2.0, but there are quite a few backward incompatible changes.

Do you have a vision on how you want to support 2.0? Are you thinking of a new major version of sqlalchemy-bigquery that only supports sqlalchemy 2.0+? If yes, this makes the patch simpler. Supporting both <2.0 and >=2.0 seems incredibly hard with the select() syntax change.

sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 26, 2023
@sharoonthomas
Add support for SQLAlchemy 2.0
098e4ca 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
@sharoonthomas sharoonthomas mentioned this issue Oct 26, 2023
Closed
sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 26, 2023
@sharoonthomas
feat: Add support for SQLAlchemy 2.0
28dfde2 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 26, 2023
@sharoonthomas
feat: Add support for SQLAlchemy 2.0
581869e 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 26, 2023
@sharoonthomas
feat: Add support for SQLAlchemy 2.0
aa4dd65 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
@tswast
Copy link
Collaborator

tswast commented Oct 26, 2023

Thanks @sharoonthomas for taking the initiative on this! I'm just a 20% contributor on this project these days, but I'll share internally with the current maintainers to review.

I would suggest a new major version for such a change. Please prefix your commit message with feat!: , such as feat!: support SQLAlchemy 2.0, drop support for 1.x. We use release-please to manage our releases & changelog: https://github.com/googleapis/release-please/blob/main/README.md#how-should-i-write-my-commits

sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 26, 2023
@sharoonthomas
feat!: Support SQLAlchemy 2.0, drop support for 1.x
99e352c 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
@sharoonthomas
Copy link
Contributor

Thank you @tswast. Updated the commit message.

sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 26, 2023
@sharoonthomas
feat!: Support SQLAlchemy 2.0, drop support for 1.x
09fc63a 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 26, 2023
@sharoonthomas
feat!: Support SQLAlchemy 2.0, drop support for 1.x
fb57dee 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
@Giuzzilla
Copy link

Giuzzilla commented Oct 27, 2023 •
edited
Loading

Hey @sharoonthomas, thanks for your work on this, just a little note: SQLAlchemy 1.4.x supports the new select syntax (it's intended to be a bridge release for 2.x). Not sure what other changes you had to do, but maybe you can consider relaxing the constraint to >= 1.4.

sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Oct 31, 2023
@sharoonthomas
feat!: Support SQLAlchemy 2.0, drop support for 1.x
55a043c 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
@sharoonthomas sharoonthomas mentioned this issue Oct 31, 2023
Closed
sharoonthomas added a commit to sharoonthomas/python-bigquery-sqlalchemy that referenced this issue Nov 7, 2023
@sharoonthomas
feat!: Support SQLAlchemy 2.0, drop support for 1.x
ffc491f 
This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes googleapis#510
@kiraksi kiraksi self-assigned this Nov 16, 2023
@kiraksi kiraksi mentioned this issue Nov 29, 2023
Closed
4 tasks
@chalmerlowe
Copy link
Collaborator

A pre-release version of python-bigquery-sqlalchemy has been released with coverage for 1.4 AND 2.0.
Gonna close this issue as complete.

As feedback on the pre-release rolls in, we will update that branch and ultimately merge with main when ready.

@Linchin Linchin mentioned this issue Feb 2, 2024
Closed
chalmerlowe added a commit that referenced this issue Apr 3, 2024
@tswast @sharoonthomas
@nayaknishant @gcf-owl-bot @kiraksi @chalmerlowe
feat: support SQLAlchemy 2.0, raise minimum required version to 1.4.x (…
7a4c3c2 
…#1053)

* feat!: Support SQLAlchemy 2.0, drop support for 1.x

This patch makes this library backward incompatible for
versions of SQLAlchemy < 2.0.

Fixes #510

* constraints updated

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* fixing README.rst

* fixing README.rst

* upping sqlalchemy version in constraints-3.8.txt

* adding 2.0 version restrictions to owlbot.py

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* fix for

* Updated some compliance tests for sqla2 and bq

* Addressed snippet errors

* revert bad commit

* More compliance tests checking

* reformatted with black

* Changed more compliance tests, updated requirements for testing

* Fixed attributeerror in failing sample test

* Fixed geolography test failing issue

* Minor tweaks to tests and code

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Fixed small error in compliance tests, added pip freeze for owl bot testing

* Fixed some failing compliance tests by reformatting

* Added UuidTest to compliance tests

* Moved back sqlalchemy constraints to 1.4

* Update testing/constraints-3.8.txt

* Fixed minimum version of sqlalchemy for 1.4 backwards compatibility

* Bumping support for sqlalchemy 1.4.16 for sample tests

* Bump setup.py sqlalchemy to 1.4.16

* Updated compliance sqlalchemy to 1.4.16

* Fixed broken code in last merged main, as we need to avoid duplicate entries and potential implicit joins due to identical table names

* modified tests for join order variation in 1.4 vs 2.0

* typo

* Modified one compliance StringTest thats been flaky

* Updated docs

* minor fixes to noxfile and README

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* cleaned up code from review, removed unnecessary code and files

* Update tests/sqlalchemy_dialect_compliance/test_dialect_compliance.py

* create development release 1.11.0.dev0 branch

* chore: cleanup compliance tests for sqlalchemy migration (#1013)

* chore: remove code for sqlalchemy before 1_4

* reformatted with black:

* Removed sqlalchemy compliance tests from versions before 1.4

* removed code in base.py for sqlalchemy < 1.4

* fix coverage issues in base.py

* temporarily commented out code lines not passing coverage for testing purposes

* replaced functions previously removed for not passing cover

* testing removing functions for coverage

* add no cover tag to untested code and clean up commented out functions

* fix lint issues

* black

* chore: cleanup compliance file tests after migration

* lint

* fixed small import error

---------

Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io>

* Revert "chore: cleanup compliance tests for sqlalchemy migration" (#1015)

* chore: fix coverage tests sqlalchemy 2.0 migration (#987)

* chore: remove code for sqlalchemy before 1_4

* reformatted with black:

* Removed sqlalchemy compliance tests from versions before 1.4

* removed code in base.py for sqlalchemy < 1.4

* fix coverage issues in base.py

* temporarily commented out code lines not passing coverage for testing purposes

* replaced functions previously removed for not passing cover

* testing removing functions for coverage

* add no cover tag to untested code and clean up commented out functions

* fix lint issues

* black

* Readded deleted tests and renamed them from deprecated names

* black

---------

Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io>

* chore: sqlalchemy test compliance suite cleanup (#1018)

* chore: remove code for sqlalchemy before 1_4

* reformatted with black:

* Removed sqlalchemy compliance tests from versions before 1.4

* removed code in base.py for sqlalchemy < 1.4

* fix coverage issues in base.py

* temporarily commented out code lines not passing coverage for testing purposes

* replaced functions previously removed for not passing cover

* testing removing functions for coverage

* add no cover tag to untested code and clean up commented out functions

* fix lint issues

* black

* Readded deleted tests and renamed them from deprecated names

* black

* chore: sqlalchemy test compliance suite cleanup code

* black

* black

---------

Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io>

* create development release 1.11.0.dev1 branch

* feat: grouping sets, rollup and cube compatibility

* create development release 1.11.0.dev2

* test commit to run kokooro tests

* removed unnecessary clause function changes, edited tests

* test basic implementation of group_by_clause and visit_label

* fixed render label as label assignment

* added test case

* reformat logic

* test commit

* create development build 1.11.0.dev3

* chore: add more grouping sets/rollup/cube tests (#1029)

* chore: add more tests for grouping functions fix

* reformatted tests

* update changelog

* revert changelog

* remove note

* don't install prerelease in compliance session

* sync owlbot

* Update tests/sqlalchemy_dialect_compliance/test_dialect_compliance.py

* make results order not matter

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io>
Co-authored-by: Nishant Nayak <nishantnayak@google.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: kiraksi <kirnendra@google.com>
Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
Co-authored-by: Chalmer Lowe <chalmer.lowe@gmail.com>
Co-authored-by: Kira <kira.kaur.sidhu@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kiraksi kiraksi

Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Milestone
No milestone