Skip to content

Conversation

@Crowiant
Copy link
Contributor

@Crowiant Crowiant commented Jul 11, 2025

  • Add ray package as a separate dependency for google provider to avoid downgrading of the setuptools package that happens if ray was installed as google-cloud-aiplatform[ray] ( https://github.com/googleapis/python-aiplatform/blob/main/setup.py#L114 ).
  • Remove default values for the fields python_version and ray_version in CreateRayClusterOperator.
  • Add documentation regarding the changes.
  • Change system test for ray operators.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Jul 11, 2025
@potiuk
Copy link
Member

potiuk commented Jul 11, 2025

Is not that contradicitng that ray package is >= 2.42 and that supported ray versions are 2.9.3, 2.33, 2.42 ?

@Crowiant
Copy link
Contributor Author

Is not that contradicitng that ray package is >= 2.42 and that supported ray versions are 2.9.3, 2.33, 2.42 ?

@potiuk I tested it with ray version 2.47(the latest) in breeze and it works ok. The only warning from vertex ai that it is better to have the same library.
It only impacts the version(field ray_version) that user can use to create a ray cluster in GCP.

@Crowiant Crowiant force-pushed the fix-ray-operators branch from 10dc080 to d77b636 Compare July 11, 2025 14:16
@VladaZakharova
Copy link
Contributor

@potiuk
Hi
We were thinking how to make it work in Google provider and in Composer, so this is the solution for now

@Crowiant Crowiant marked this pull request as ready for review July 11, 2025 14:49
@potiuk potiuk merged commit bc79040 into apache:main Jul 11, 2025
142 of 158 checks passed
stephen-bracken pushed a commit to stephen-bracken/airflow that referenced this pull request Jul 15, 2025
…ncy. (apache#53182)

Co-authored-by: Anton Nitochkin <nitochkin@google.com>
Comment on lines +759 to +761
Please note that you need to specify python_version and ray_version in :class:`~airflow.providers.google.cloud.operators.vertex_ai.ray.CreateRayClusterOperator`.
Currently supported versions of ray package in ray cluster are: 2.9.3, 2.33, 2.42.
For more information you can check: https://github.com/googleapis/python-aiplatform/blob/main/setup.py#L101
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like that our docs reference to internal details of other packages - especially when we are referencing main branch where any commit can make this link not accurate. doesn't the library serve a user facing doc that we can reference instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand, this is specifically for users to see the limitations of the package they are trying to use, because without this it will be hard to determine what is wrong if the user will use the operator with unsupported versions of Ray.
For the link to the comment I agree, maybe better to refer to it in another way, more flexible one

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The package has doc it looks like they just don't document which ray version they support. Might worth asking them to do so?
The sentence of

Currently supported versions of ray package in ray cluster are: 2.9.3, 2.33, 2.42.
For more information you can check: https://github.com/googleapis/python-aiplatform/blob/main/setup.py#L101

Should be something like:
To read more about compatible versions of Ray check the Google docs in <Link to upstream doc>

This avoids the need to update our docs for every change upstream library makes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants