BLD: add optional dependencies as extras_require in setup.py

[JMBurley]

EDIT: This was originally a limited scope PR to bring recommended dependencies into pandas extras_require, but discussion evolved into 'solve packaging for all optional dependencies' rather than just solve for the recommended optional dependencies. The logic behind this is the same as originally argued for recommended dependencies but I updated the text of the post to keep it aligned with the PR contents

---

Closes #47335. Closes #39164 . Closes #48361 .

Hoping to make this part of the 1.5 milestone.

Optional dependencies should have package mgmt facilitated through pandas. This will make mgmt of pandas in production docker environments a significantly simpler and allow pandas explicit control over the versions of optional packages it should operate with.

Actions

Use setup.cfg options as documented here to make optional extras for a pandas install.

Once this is done, a user will be able to ensure version alignment by

# in requirements.txt or setup.py
pandas[recommended, s3] == 1.22.4  # install pd 1.22.4 and compatible numexpr, bottleneck, s3fs versions

The proposed solution is simple to do, simple to maintain, and already done by major libraries in similar situations (moto, aiobotocore).

Not having this solution wastes user time by forcing them to manually maintain versions of pandas dependencies that are not clearly linked to pandas, save in the brains of users and by manually referring to an install docs page.

checklist

• closes ENH: recommended dependencies numexpr and bottleneck should be managed extras in pandas setup.cfg #47335 , closes ENH: add additional extras_require sections for optional dependencies #39164
• [NA] Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• [NA] Added type annotations to new arguments/methods/functions. (no new functions)
• Added an entry in the latest doc/source/whatsnew/v1.4.3.rst file if fixing a bug or adding a new feature. (in Other section)

[JMBurley]

Test failure Windows-MacOS / windows-latest actions-39.yaml (pull_request) seemingly unrelated to modified code.

This same test fails in PRs that are approved to be merged so unclear to me if action is required.

INTERNALERROR> AssertionError: ('pandas/tests/plotting/frame/test_frame_color.py::TestDataFrameColor::test_mpl2_color_cycle_str', <WorkerController gw0>)
INTERNALERROR> assert not 'pandas/tests/plotting/frame/test_frame_color.py::TestDataFrameColor::test_mpl2_color_cycle_str'

[jbrockmendel]

This same test fails in PRs that are approved to be merged so unclear to me if action is required.

Unrelated, don't worry about it.

[mroeschke]

As noted in #39164 (comment), I think the correct groupings should be discussed first before creating a pull request as it would be good to get all the optional dependencies in

[JMBurley]

Updates made as per discussion in #39164

[JMBurley]

@mroeschke Ready for review.

[mroeschke]

I would personally like to see this be called filesystem as fss isn't immediately obvious to me, but not a blocker to me.

[JMBurley]

It's a good question. I went back and forth on fss vs filesystem. Ultimately I decided that filesystem might suggest to users a broad compatibility with common filesystems (eg. s3) that the optional_extra does not provide, and it was probably least surprising to users to call it fss. (and the alternative fsspec is worryingly specific in case fsspec were ever deprecated and pandas switched to an alternative filesystem mgmt package in future).

But I don't think either solution is perfect and am very open to alternative suggestions or being argued back to filesystem.

[mroeschke]

Gotcha, yeah fss would be okay as well I suppose.

[mroeschke]

Great thanks for your work and patience here, LGTM. Will leave open just in case others want to chime in

[JMBurley]

Great thanks for your work and patience here, LGTM. Will leave open just in case others want to chime in

Awesome, glad to have this in. Appreciate the reviews to keep this PR on track. Optional dependencies are really helpful for container mgmt in production DS.

Note: there is now a conflict with main branch where whatsnew/v1.6.0.rst has been renamed to whatsnew/v2.0.0.rst. Is pandas 2.0.0 now the next major release instead of pandas 1.6.0?

I'll let someone provide guidance before I alter documentation for the merge.

[mroeschke]

Is pandas 2.0.0 now the next major release instead of pandas 1.6.0?

Correct, 2.0 will be the next major release so a rebase is needed

[JMBurley]

Approved and tests passed!
First enhancement for pandas 1.6.0 in the bag!
First enhancement for pandas 2.0.0 in the bag!

[mroeschke]

Sorry this PR hasn't gotten that much other review @JMBurley. I think this will be okay to merge once the conflict is resolved (the pyarrow min version was bumped to 6.0)

[JMBurley]

Sorry this PR hasn't gotten that much other review @JMBurley. I think this will be okay to merge once the conflict is resolved (the pyarrow min version was bumped to 6.0)

Pyarrow version is now bumped & merged in latest from main. Waiting for tests to pass

[JMBurley]

@mroeschke good to merge?

[mroeschke]

Great work @JMBurley! Thanks for sticking with all the requests. I do think this is really beneficial with all the optional dependencies pandas!

[Dr-Irv]

@mroeschke I found this part confusing:

Optional pandas dependencies can be managed as optional extras (e.g.,``pandas[performance, aws]>=1.5.0``)
in a requirements.txt, setup, or pyproject.toml file.
Available optional dependencies are ``[all, performance, computation, aws,
gcp, excel, parquet, feather, hdf5, spss, postgresql, mysql, sql-other, html, xml,
plot, output_formatting, compression, test]``

I think this should point out that the optional dependencies should ONLY be managed via files, and that you cannot do conda install pandas[computation]=1.5.1 (for example) to get the dependencies to install. The way this section is written, it makes it seem that you can do that (or maybe you can with this PR, in which case that should be shown as an installation option).

[mroeschke]

Good points @Dr-Irv . Yeah this type of install should be only valid with pip installs (command line & files), so I can follow up clarifying the documentation and also give an installation example.

[jorisvandenbossche]

Echoing @Dr-Irv's comment, also here I find this confusing, since at the top of this document we actually start with mentioning conda install pandas before pip install pandas. So I would personally remove the pandas[performance] here, or at least every time we mention such an extra also point to a dedicated section explaining it?

[JMBurley]

I think this should point out that the optional dependencies should ONLY be managed via files, and that you cannot do conda install pandas[computation]=1.5.1 (for example) to get the dependencies to install. The way this section is written, it makes it seem that you can do that [in conda] (or maybe you can with this PR, in which case that should be shown as an installation option).

Good point. As far as I'm aware there is absolutely no way replicate optional dependencies in conda, and the workaround would be to make individual recipes with individual names for every conceivable combination of optional_extras.

we actually start with mentioning conda install pandas before pip install pandas.

I wonder if pandas mentioning condas first is a side-effect of pandas being extant from pre-pip and enduring pip's early days when conda had significant advantages. The python ecosystem has pip-first since python 3.4 while we should certainly support both, I think pip should be the primary install suggestion for pandas.

Regardless, I agree that having a single section explaining how to install extras and then continually linking to it from elsewhere in installs.rst is an improvement.