Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Drop Python 3.7 #213

Closed
jrbourbeau opened this issue Jan 14, 2022 · 17 comments
Closed

Drop Python 3.7 #213

jrbourbeau opened this issue Jan 14, 2022 · 17 comments

Comments

@jrbourbeau
Copy link
Member

I'd like to propose that Dask drops support for Python 3.7. NEP 29 (which there was general interest in for Dask #66) states that support for Python 3.7 can be dropped starting on Dec 26, 2021. Additionally, other projects in the ecosystem (e.g. NumPy, Xarray) have already begun dropping support for Python 3.7. I've brought this up offline with a group of Dask maintainers and they didn't have any objections, but wanted to raise an issue here to solicit feedback from others.

cc @jsignell @quasiben @jakirkham @fjetter @crusaderky @ian-r-rose
@jakirkham
Copy link
Member

Generally this sounds like a good idea and should simplify some things in Dask + Distributed.

That said, there's at least a couple issues I can think of atm.

First is PyPy hasn't quite made it to 3.8 yet. It sounds like it is close, but may need a few more weeks based on Matti's recent summary here ( conda-forge/conda-forge-pinning-feedstock#2089 (comment) ). Related to that, in conda-forge we are discussing building packages for PyPy 3.8, but this likely needs to wait until things are completed in PyPy as well as for the existing migrations to clear (like Python 3.11). So may take a bit more time before packages are built for PyPy 3.8.

Second RAPIDS currently still supports Python 3.7. Not exactly sure when we plan to drop that. Maybe Ben can comment.

Third there are still users/providers on Python 3.7 that haven't yet made the move. Believe Kaggle is one. Not sure who else.

@jacobtomlinson jacobtomlinson mentioned this issue Jan 18, 2022
Closed
@jrbourbeau
Copy link
Member Author

Thanks for the feedback @jakirkham!

Second RAPIDS currently still supports Python 3.7. Not exactly sure when we plan to drop that. Maybe Ben can comment.

@quasiben do you have any insight into when RAPIDS is planning to drop Python 3.7? FWIW it looks like there's some preparation for this which is already underway rapidsai/cudf#10092

First is PyPy hasn't quite made it to 3.8 yet. It sounds like it is close, but may need a few more weeks based on Matti's recent summary here ( conda-forge/conda-forge-pinning-feedstock#2089 (comment) )

That's a fair point and it'd be good for us to be concious of pypy users. That said, with major dependencies like NumPy (latest release is Python 3.8+) and pandas (upcoming 1.4 release is Python 3.8+) dropping support for Python 3.7, it will make it harder for Dask to continue to support Python 3.7. @mattip, thank you for pushing on pypy3.8. Do you have an estimate for when a release might be?

@mattip
Copy link

mattip commented Jan 21, 2022

It should be done within a month. I want to get a release candidate out next week, covid and flu willing. If dask does drop Python3.7, that only means new versions of dask would not be available, right? It is not as if you would yank the older packages.

@quasiben
Copy link
Member

I think the plan is to drop 3.7 in our upcoming release if we are successful at getting the 3.9 builds working across all RAPIDS libraries.

@quasiben
Copy link
Member

FYI deprecation notice is here: https://docs.rapids.ai/notices/rsn0013/

@jrbourbeau jrbourbeau mentioned this issue Jan 21, 2022
Merged
@jakirkham jakirkham mentioned this issue Jan 21, 2022
Merged
@jakirkham
Copy link
Member

If dask does drop Python3.7, that only means new versions of dask would not be available, right? It is not as if you would yank the older packages.

That's right, Matti

@quasiben
Copy link
Member

quasiben commented Jan 31, 2022 •
edited
Loading

@jrbourbeau the rapidsai-nightly channel now has 3.9 packages and will be continuing down the path of dropping 3.7 support

@quasiben
Copy link
Member

quasiben commented Feb 3, 2022

I believe RAPIDS has now dropped support for 3.7. All new packages are for published for Python 3.8 and 3.9 in https://anaconda.org/rapidsai/rapids/files and https://anaconda.org/rapidsai-nightly/rapids/files

@douglasdavis douglasdavis mentioned this issue Feb 7, 2022
Merged
@davidhassell davidhassell mentioned this issue Feb 9, 2022
Merged
@jrbourbeau
Copy link
Member Author

Thanks for the update @quasiben -- I'll suggest we go ahead and drop Python 3.7 then

@jsignell
Copy link
Member

Time to merge the PRs?

@jrbourbeau
Copy link
Member Author

Let's get an approval on dask/distributed#5683 first. But then yeah, dask/distributed#5683 and dask/dask#8572 should go in together

@jakirkham
Copy link
Member

Oh are they done/ready for review?

@jacobtomlinson jacobtomlinson mentioned this issue Feb 16, 2022
Merged
@jakirkham
Copy link
Member

This is now complete for Dask + Distributed. Possibly other packages will need similar changes, but that can be followed up on separately. Going to go ahead and close as resolved.

@jacobtomlinson
Copy link
Member

This has been done in dask-kubernetes too.

@jacobtomlinson
Copy link
Member

I have noticed that Read the Docs still has Python 3.7 as the default for building documentation and that is causing build errors in places since Dask 2022.2.1 was released. I've raised dask/dask-kubernetes/pull/412 to bump to 3.8 (the highest Read the Docs allows right now) resolve things there. I suspect this will need doing on all projects that use Read the Docs.

@jakirkham
Copy link
Member

There is a way to override this in a .readthedocs.yaml config file. Maybe this is also overridable in the ReadTheDocs settings?

FWIW we discussed adding a .readthedocs.yaml config file in issue ( dask/dask#2568 ) and wound up doing this when dropping Python 3.7 ( dask/dask#8572 ).

@jacobtomlinson
Copy link
Member

There is a way to override this in a .readthedocs.yaml config file. Maybe this is also overridable in the ReadTheDocs settings?

I couldn't see anything in the UI to set this.

I've added a .readthedocs.yaml file to dask-kubernetes and that resolved the issue. But this will need to be done for all projects that use Dask/Distributed.

This was referenced Mar 25, 2022
Closed
Merged
jsignell pushed a commit to dask/dask that referenced this issue May 6, 2022
@douglasdavis
Collection Protocol (#8674)
1e783d9 
[PEP 544](https://www.python.org/dev/peps/pep-0544/) introduces the `Protocol` class to the `typing` module in Python 3.8 (the soon be the minimum supported version, dask/community#213). Writing new Dask collections for [dask-awkward](https://github.com/ContinuumIO/dask-awkward/) has had me thinking about working on a `DaskCollection` protocol. I imagine the benefits to be:

- usage with static type checkers
  - other activity in this area at
    - #8295 
    - #8706 
    -  #8854
  - Python supporting IDEs take advantage of typing
- self-documenting; some improvements to [the custom collections page](https://docs.dask.org/en/latest/custom-collections.html) of the docs. The protocol docs can be autogenerated and added to that page.
- purely opt-in feature

The `typing.runtime_checkable` decorator allows use of `isinstance(x, DaskCollection)` in any code base
that uses Dask collections; for example:

```python
>>> from dask.typing import DaskCollection
>>> import dask.array as da
>>> x = da.zeros((10, 3))
>>> isinstance(x, DaskCollection)
True
```
(though this is an order of magnitude slower than `dask.base.is_dask_collection` which only checks for `x.__dask_graph__() is not None`; static typing checking & built-in interface documentation are the core benefits IMO)

Something else that came up in the brief discussion on a call last week was having `{Scheduler,Worker,Nanny}Plugin` protocols in `distributed`; and perhaps those are better places to start introducing protocols to Dask since on the user side typically more folks would write plugins than new collections.
erayaslan pushed a commit to erayaslan/dask that referenced this issue May 12, 2022
@douglasdavis @erayaslan
Collection Protocol (dask#8674)
2aab6bf 
[PEP 544](https://www.python.org/dev/peps/pep-0544/) introduces the `Protocol` class to the `typing` module in Python 3.8 (the soon be the minimum supported version, dask/community#213). Writing new Dask collections for [dask-awkward](https://github.com/ContinuumIO/dask-awkward/) has had me thinking about working on a `DaskCollection` protocol. I imagine the benefits to be:

- usage with static type checkers
  - other activity in this area at
    - dask#8295 
    - dask#8706 
    -  dask#8854
  - Python supporting IDEs take advantage of typing
- self-documenting; some improvements to [the custom collections page](https://docs.dask.org/en/latest/custom-collections.html) of the docs. The protocol docs can be autogenerated and added to that page.
- purely opt-in feature

The `typing.runtime_checkable` decorator allows use of `isinstance(x, DaskCollection)` in any code base
that uses Dask collections; for example:

```python
>>> from dask.typing import DaskCollection
>>> import dask.array as da
>>> x = da.zeros((10, 3))
>>> isinstance(x, DaskCollection)
True
```
(though this is an order of magnitude slower than `dask.base.is_dask_collection` which only checks for `x.__dask_graph__() is not None`; static typing checking & built-in interface documentation are the core benefits IMO)

Something else that came up in the brief discussion on a call last week was having `{Scheduler,Worker,Nanny}Plugin` protocols in `distributed`; and perhaps those are better places to start introducing protocols to Dask since on the user side typically more folks would write plugins than new collections.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@mattip @quasiben @jacobtomlinson @jakirkham @jsignell @jrbourbeau