Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Bumping pyarrow / pandas #17290

Merged
merged 5 commits into from
Oct 31, 2021
Merged

Conversation

craig-rueda
Copy link
Member

@craig-rueda craig-rueda commented Oct 29, 2021

SUMMARY

Bumps pandas to 1.3.4 and pyarrow to 5.0.0. Note that 6.0.0 is the latest pyarrow, but there are other dependencies with conflicting ranges. Also note that pylint yelled at me about various superfluous "ignores" that I ended up fixing. Hence all the seemingly unrelated changes.

@pull-request-size pull-request-size bot added size/M and removed size/L labels Oct 29, 2021
@codecov
Copy link

codecov bot commented Oct 30, 2021

Codecov Report

Merging #17290 (5d2f7a3) into master (b7bda70) will increase coverage by 0.00%.
The diff coverage is 75.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #17290   +/-   ##
=======================================
  Coverage   77.08%   77.08%           
=======================================
  Files        1037     1037           
  Lines       55653    55654    +1     
  Branches     7607     7607           
=======================================
+ Hits        42901    42902    +1     
  Misses      12502    12502           
  Partials      250      250           
Flag Coverage Δ
hive 81.55% <75.00%> (+<0.01%) ⬆️
mysql 81.97% <75.00%> (+<0.01%) ⬆️
postgres 81.98% <75.00%> (+<0.01%) ⬆️
presto 81.84% <75.00%> (+<0.01%) ⬆️
python 82.48% <75.00%> (+<0.01%) ⬆️
sqlite 81.66% <75.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engine_specs/bigquery.py 84.84% <ø> (ø)
superset/db_engine_specs/druid.py 86.27% <ø> (ø)
superset/db_engine_specs/snowflake.py 76.10% <ø> (ø)
superset/config.py 91.50% <50.00%> (ø)
superset/sql_lab.py 80.85% <100.00%> (ø)
superset/utils/core.py 90.14% <100.00%> (ø)
superset/views/core.py 76.95% <0.00%> (+0.01%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7bda70...5d2f7a3. Read the comment docs.

Copy link
Member

@zhaoyongjie zhaoyongjie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@craig-rueda craig-rueda merged commit 6a63a21 into apache:master Oct 31, 2021
@craig-rueda craig-rueda deleted the craig/bump-pyarrow branch October 31, 2021 17:47
@cwegener
Copy link
Contributor

pyarrow 6.0 is the minimum requirement to use the python 3.10 wheel for pyarrow.
Without pyarrow 6.0, installing superset under python 3.10 becomes almost impossible due to the difficulty of compiling pyarrow from source.

I can upgrade easily to pyarrow 6.0 with only a few dependency changes (thereby allow me to run superset with python 3.10):

  1. Remove pinning on pypackaging 21.0 (thereby allowing 21.3)
  2. Remove version block that blocks pyparsing version upgrades to 3.0 and higher (thereby allowing pyparsing 3.0.6)
  3. Remove pinning on pylint 2.9.6 (thereby allowing 2.12.2)

@craig-rueda Do you see any problems with the above changes?

The upgrade block to pyparsing 3.0 was introduced in #11418 by @zhaoyongjie . I suspect that a review and some testing of the time picker feature is required to assess compatibility with pyparsing 3.0.6

@craig-rueda
Copy link
Member Author

I think it should be fine. I'll test it out on M1 hardware to be certain in the next day or so. That's the only potential issue I know of.

@cwegener
Copy link
Contributor

FIY, I have been running a local dev set up (no docker, just plain python 3.10 on Arch Linux) without problem with the above changes. I am certainly not claiming to have done any sort of rigorous testing.

Let me know if you would like me to open an issue / PR for this dependency update change.

AAfghahi pushed a commit that referenced this pull request Jan 10, 2022
* Bumping pyarrow / pandas

* Appeasing the pylint gods

* Adjusting UPDATING.md

* Buping testing requirements
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels preset-io size/M 🚢 1.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants