Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy #53392
Replies: 4 comments 7 replies
-
|
I am not sur where this error comes from ? How do you think it comes from those version and not your wrong usage? the cohnstraints in airflow are generated when there is no error in unit tests of airflow showing that those versions work. As far as I know the issue you have is because some code you are using needs sqlalchemy 2: Which waits for FAB 5 to be merged: Without knowing what exactly caused your issue, we are unable to spend our time on trying to guess what is causing your problem, but if you can spend some time on providing more details, complete stack traces and more context, we might. |
Beta Was this translation helpful? Give feedback.
-
|
pandas 2.2.3 has the optional requirement for SQLAlchemy >2.0.0 however, this workaround seems to work right now: pandas-dev/pandas#57049 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
I have seen an issue on great-expectations and I guess it could contribute to this discussion. They have dependencies on Pandas, SQLAlchemy and Airflow and one suggestion is a fine tuning on Airflow requirements. I don't know if it is reasonable, my intention was only to bring it to your attention.
Originally commented by @jonaslb in great-expectations issue #10621 |
Beta Was this translation helpful? Give feedback.
-
|
converting to a discussion as more discussion and input from reporter is needed. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have the same issue. All tasks that have the df.to_sql() method failed with the same error in airflow v3.0.3. I had to downgrade to 3.0.2 and everything started working again. |
Beta Was this translation helpful? Give feedback.
-
|
@potiuk the airflow constraints file is wrong however: https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt it has: pandas==2.2.3 These are not compatible as pandas >=2.2.0 only supports sqlalchemy>=2.0.0 |
Beta Was this translation helpful? Give feedback.
-
Technically the two packages are compatible when "base" version of those packages are installed. From what I see - they are only compatible with sqlalchemy 2.2.0 for "postgres", "mysql", "sql-other" and "all" extras So you can install together I see the issue though. I will take a look and see if we should some of the extras in our pandas requirements. That should fix the problem in future versions. In the meantime feel free to just downgrade pandas. You can also copy the constraints of airlfow, replace pandas version there and use them instead. |
Beta Was this translation helpful? Give feedback.
-
|
PR here: #53535 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @jabbera for following up with this one despite me being hesitant. It turned out that that our pandas requirement for postgres and common.sql providers just missed those extras - that was the root of the problem. Yeah. The pr worked. You will see in "source providers" constraints in main that pandas is now 2.1.4. When the next wave of providers is released (next week) also "constraints-3.11.txt" will be updated (those constraints are build automatically based on the released providers in PyPI (actually in main it should get updated as soon as RCs are released - because our constraint building mechanism includes also RC candidates as of few months). Then, when we release 3.0.4 it will use the released providers, and then it should get update for 3.0.4 constraints. Sometimes we fix constraints for past released versions (last time yesterday) - https://airflow.apache.org/docs/docker-stack/changelog.html#changes-after-publishing-the-images -> but we only do it for really critical changes that affect many users, in this case I think the issue is really affecting small number of users who want to use from_df/to_df and there is an existing workaround (just downgrade pandas to 2.1.4) so I do not see a need to regenerate the images and update constraints manually. Unless there are many people complaining and some other reasons why this is problematic to have pandas 2.2.3 in constraints. |
Beta Was this translation helpful? Give feedback.
-
|
I also converted the discussion back to issue and posted detailed explanation of the problem and explanation on workarounds you might use: #53560 Let me know if you think this is enough for you: @jacobeulogio @aa-matthias @jabbera @oailton and if you are fine with the workarounds posted. |
Beta Was this translation helpful? Give feedback.
-
|
UPDATE: in order to have consistent constraints, you need to update:
Otherwise numpy will generate conflict. I updated the workaround description in the issue. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Apache Airflow version
3.0.3
If "Other Airflow 2 version" selected, which one?
No response
What happened?
constraints for 3.0.3 have pandas 2.2.3 and SQLAlchemy 1.4.54 causing this issue now:
pandas-dev/pandas#57049
What you think should happen instead?
No response
How to reproduce
Upgrade to 3.0.3 using constraints for 3.0.3 python3.12
Operating System
Ubuntu 24.04
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions