Error when building from source: the existence of python/ray/types.py causes an ImportError

[sushant-t]

What happened + What you expected to happen

I am following the "Building Ray (Python Only)" section of the documentation. I have created my Conda environment, and I have installed the latest Ray wheel. When I run python python/ray/setup-dev.py, I get the following error:

Traceback (most recent call last): File "/Users/st/ray/python/ray/setup-dev.py", line 6, in <module> import argparse File "/Users/st/opt/anaconda3/envs/ray/lib/python3.10/argparse.py", line 89, in <module> import re as _re File "/Users/st/opt/anaconda3/envs/ray/lib/python3.10/re.py", line 124, in <module> import enum File "/Users/st/opt/anaconda3/envs/ray/lib/python3.10/enum.py", line 2, in <module> from types import MappingProxyType, DynamicClassAttribute File "/Users/st/ray/python/ray/types.py", line 1, in <module> from typing import Generic, TypeVar File "/Users/st/opt/anaconda3/envs/ray/lib/python3.10/typing.py", line 25, in <module> import contextlib File "/Users/st/opt/anaconda3/envs/ray/lib/python3.10/contextlib.py", line 6, in <module> from functools import wraps File "/Users/st/opt/anaconda3/envs/ray/lib/python3.10/functools.py", line 22, in <module> from types import GenericAlias ImportError: cannot import name 'GenericAlias' from partially initialized module 'types' (most likely due to a circular import) (/Users/st/ray/python/ray/types.py)

I figured out that this error comes because there is a types.py file in the same directory as the setup-dev.py. Thus, Python's dependency resolution module gets confused and tries to import GenericAlias from the types.py file in the same directory, instead of the actual types module.

The solution is simply to rename the types.py file to anything else, and the setup-dev.py file works as intended.

Has this been addressed already? If not, I would be happy to discuss and implement a solution (or at the least update the docs accordingly) so that any future users of Ray do not run into this issue.

Versions / Dependencies

conda version: 22.11.1
ray version: 3.0.0.dev0
python version: 3.10.8

Using MacOS Monterey 12.6.1, Apple Silicon

Reproduction script

conda create -n ray python=3.10
conda activate ray
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-macosx_10_15_universal2.whl
cd ray
python python/ray/setup-dev.py

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[krfricke]

This is an interesting error. E.g. it does come up in Python 3.8.6, but not in Python 3.8.7. It seems to come up for all Python 3.10 versions I've tried.

It can be triggered by just going into ray/python/ray, starting a python shell and running import argparse.

TBH, this looks like a python bug to me - global packages should not try local imports. However, the reality seems to be that this is the case and we have to deal with it across multiple python versions.

I've submitted a bug to python here: python/cpython#101210

As for fixing this issue, I'll see if we can do this without renaming types.py as it's a public API and we don't want to deprecate it. Another option would be to just move setup-dev.py, so we may do that.

[krfricke]

@sushant-t there is a fix here: #31829 can you take a look and confirm this fixes your issue?

[sushant-t]

@krfricke perfect, that fixed it! I reproduced the above steps with your commit and the setup-dev.py runs with no issues.

[RaffaGonzo]

I ran into a similar and related issue with a different import in setup-dev.py. For me though, I was using Python 3.8.3 and all of these fixes were for Python 3.8.6 and above if I interpret this correctly. Anyway, what worked for me was to update my version of Python to 3.9.18 and use that for my virtual environment.