Ray is not installed via google-cloud-aiplatform[ray] on 3.12

[insomnes]

After the recent addition of vertex ai operators to Apache Airflow there are problems now with tests on only python 3.12, because ray extra is not installed for 3.12.

I suppose the main problem is that ray extra of google-cloud-aiplatform has ray[default] options for ==3.11 and <3.11, so it's not installing ray on > 3.11:

https://github.com/googleapis/python-aiplatform/blob/main/setup.py#L103C1-L103C18

Environment details

• OS type and version: Linux
• Python version: Python 3.12.7
• pip version:
• google-cloud-aiplatform version: 1.73.0, also checked on 1.80.0, 1.90.0

Steps to reproduce

1. Python 3.12 environment
2. Install google-cloud-aiplatform[ray]
3. Try to import anything with ray import like from google.cloud.aiplatform.vertex_ray.util import resources

docker run -it --rm python:3.12 /bin/bash -c 'pip install google-cloud-aiplatform[ray] && python -c "from google.cloud.aiplatform.vertex_ray.util import resources"'

Compare it to 3.11:

docker run -it --rm python:3.11 /bin/bash -c 'pip install google-cloud-aiplatform[ray] && python -c "from google.cloud.aiplatform.vertex_ray.util import resources"'

Code example

from google.cloud.aiplatform.vertex_ray.util import resources

Stack trace

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.12/site-packages/google/cloud/aiplatform/vertex_ray/__init__.py", line 21, in <module>
    from google.cloud.aiplatform.vertex_ray.bigquery_datasource import (
  File "/usr/local/lib/python3.12/site-packages/google/cloud/aiplatform/vertex_ray/bigquery_datasource.py", line 27, in <module>
    from ray.data.block import Block
ModuleNotFoundError: No module named 'ray'

[VladaZakharova]

@matthew29tang Hi there
Can i ask, why did we close this issue? I think the problem still exists

[potiuk]

Yeah @matthew29tang - clearly there are still inconsistencies in the main version in setup.py/

the library clearly supports Python 3.12:

    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "Operating System :: OS Independent",
        "Programming Language :: Python",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.9",
        "Programming Language :: Python :: 3.10",
        "Programming Language :: Python :: 3.11",
        "Programming Language :: Python :: 3.12",
        "Topic :: Internet",
        "Topic :: Software Development :: Libraries :: Python Modules",
    ],

Yet, ray is not a dependency for Python 3.12 when ray extra is used (added !!! ## comments to emphasise the issue):

ray_extra_require = [
    # Cluster only supports 2.9.3, 2.33.0, 2.42.0 and 2.47.1. Keep 2.4.0 for our
    # testing environment.
    # Note that testing is submitting a job in a cluster with Ray 2.9.3 remotely.

   #### !!!! Only installed on Python < 3.11 !!!!! 

    (
        "ray[default] >= 2.4, <= 2.42.0,!= 2.5.*,!= 2.6.*,!= 2.7.*,!="
        " 2.8.*,!=2.9.0,!=2.9.1,!=2.9.2, !=2.10.*, !=2.11.*, !=2.12.*, !=2.13.*, !="
        " 2.14.*, !=2.15.*, !=2.16.*, !=2.17.*, !=2.18.*, !=2.19.*, !=2.20.*, !="
        " 2.21.*, !=2.22.*, !=2.23.*, !=2.24.*, !=2.25.*, !=2.26.*, !=2.27.*, !="
        " 2.28.*, !=2.29.*, !=2.30.*, !=2.31.*, !=2.32.*, !=2.34.*, !=2.35.*, !="
        " 2.36.*, !=2.37.*, !=2.38.*, !=2.39.*, !=2.40.*, !=2.41.*;"
        " python_version<'3.11'"
    ),   


   #### !!!! Only installed on Python == 3.11 !!!!! 

    # Ray Data v2.4 in Python 3.11 is broken, but got fixed in Ray v2.5.
    "ray[default] >= 2.5, <= 2.47.1; python_version=='3.11'",

   #### !!! On Python 3.12 (and above), ray is not installed

    "google-cloud-bigquery-storage",
    "google-cloud-bigquery",
    "pandas >= 1.0.0",
    "pyarrow >= 6.0.1",
    "immutabledict",
]

Is it possible to re-open the issue and fix it in upcoming version ? It seems that anyone using python 3.12 will have problem when using the library with ray extra as they will miss .... ray.

Or am I wrong @matthew29tang ?