Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#228: first cut at supporting Pandas 2.0+ #345

Merged
merged 3 commits into from
Feb 28, 2024
Merged

#228: first cut at supporting Pandas 2.0+ #345

merged 3 commits into from
Feb 28, 2024

Conversation

DavidRothblum
Copy link
Contributor

Addresses the necessary changes when upgrading to Pandas 2.0+.

The farthest reaching breaking change appears to be in _check_is_datetime_type, which has been changed to use the new pd.api.types methods.

A lot of the changes here address FutureWarnings or DeprecationWarnings:
- ‘H’ and ’T’ frequency strings are deprecated -> Use ‘h’ or ‘min’
- Passing bytes to pd.read_excel is deprecated -> Wrap with io.BytesIO
- DataFrame.applymap deprecated -> Use DataFrame.map
- Pandas/Tabula error=“ignore” deprecated -> Handle pd.to_numeric errors explicitly
- Pandas set on copy of data frame -> replace .fillna(_, inplace=True) reference

@DavidRothblum
Copy link
Contributor Author

Fixes error reported in #227:

In [1]: import gridstatus

In [2]: import pandas as pd

In [3]: pd.__version__
Out[3]: '2.2.1'

In [4]: iso = gridstatus.Ercot()

In [5]: iso.get_rtm_spp(2023)
Out[5]: 
                            Time            Interval Start              Interval End    Location Location Type            Market    SPP
0      2023-01-01 00:00:00-06:00 2023-01-01 00:00:00-06:00 2023-01-01 00:15:00-06:00   HB_BUSAVG   Trading Hub  REAL_TIME_15_MIN  -2.56
1      2023-01-01 00:00:00-06:00 2023-01-01 00:00:00-06:00 2023-01-01 00:15:00-06:00  HB_HOUSTON   Trading Hub  REAL_TIME_15_MIN  -2.56
2      2023-01-01 00:00:00-06:00 2023-01-01 00:00:00-06:00 2023-01-01 00:15:00-06:00    LZ_RAYBN     Load Zone  REAL_TIME_15_MIN  -2.56
3      2023-01-01 00:00:00-06:00 2023-01-01 00:00:00-06:00 2023-01-01 00:15:00-06:00     LZ_WEST     Load Zone  REAL_TIME_15_MIN  -2.56
4      2023-01-01 00:00:00-06:00 2023-01-01 00:00:00-06:00 2023-01-01 00:15:00-06:00   HB_HUBAVG   Trading Hub  REAL_TIME_15_MIN  -2.56
...                          ...                       ...                       ...         ...           ...               ...    ...
805915 2023-12-31 23:45:00-06:00 2023-12-31 23:45:00-06:00 2024-01-01 00:00:00-06:00    LZ_SOUTH     Load Zone  REAL_TIME_15_MIN   4.66
805916 2023-12-31 23:45:00-06:00 2023-12-31 23:45:00-06:00 2024-01-01 00:00:00-06:00     LZ_WEST     Load Zone  REAL_TIME_15_MIN  15.32
805917 2023-12-31 23:45:00-06:00 2023-12-31 23:45:00-06:00 2024-01-01 00:00:00-06:00      LZ_AEN     Load Zone  REAL_TIME_15_MIN  13.72
805918 2023-12-31 23:45:00-06:00 2023-12-31 23:45:00-06:00 2024-01-01 00:00:00-06:00    LZ_RAYBN     Load Zone  REAL_TIME_15_MIN  13.92
805919 2023-12-31 23:45:00-06:00 2023-12-31 23:45:00-06:00 2024-01-01 00:00:00-06:00     LZ_WEST     Load Zone  REAL_TIME_15_MIN  15.32

[805920 rows x 7 columns]

@DavidRothblum
Copy link
Contributor Author

DavidRothblum commented Feb 27, 2024

Looks like the docs are failing to build, but work locally with make docs. I don't have much familiarity with readthedocs/sphinx, so would appreciate some tips on how to fix!

df = pd.DataFrame(data["data"][day])
df_transformed = df.apply(
lambda col: col.apply(
lambda x: x.get('gen') if isinstance(x, dict) else pd.NA
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ruff linter is complaining this needs to be "gen".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦 ... I see you guys use black formatting 🎉 . Fixed the formatting with black on the latest commit!

@WillKoehrsen
Copy link
Collaborator

Looks like the docs are failing to build, but work locally with make docs. I don't have much familiarity with readthedocs/sphinx, so would appreciate some tips on how to fix!

I don't entirely understand the readthedocs build process. The GitHub action that builds the docs did succeed.

The output from the readthedocs build says

Could not import extension myst_nb (exception: No module named 'myst_nb')

but myst-nb should be installed since it's a requirement in pyproject.toml.

@WillKoehrsen

This comment was marked as outdated.

@WillKoehrsen
Copy link
Collaborator

WillKoehrsen commented Feb 28, 2024

I fixed the RTD build issue in another PR: #346. If you want, feel free to move those changes into this PR and I'll close that one.

Update: that PR is merged so you can merge or rebase those changes into this branch to fix the RTD issue.

Copy link

codecov bot commented Feb 28, 2024

Codecov Report

Attention: Patch coverage is 96.29630% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 91.83%. Comparing base (5b88faa) to head (75d5f81).

Files Patch % Lines
gridstatus/caiso.py 85.71% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #345      +/-   ##
==========================================
- Coverage   91.92%   91.83%   -0.10%     
==========================================
  Files          40       40              
  Lines        5935     5940       +5     
==========================================
- Hits         5456     5455       -1     
- Misses        479      485       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@WillKoehrsen WillKoehrsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks a lot for taking this on!

@WillKoehrsen WillKoehrsen merged commit 4008b6f into gridstatus:main Feb 28, 2024
10 of 11 checks passed
@DavidRothblum DavidRothblum deleted the feature/2024-02-26-pandas_2_upgrade branch February 28, 2024 22:07
@WillKoehrsen WillKoehrsen mentioned this pull request Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants