Skip to content

Conversation

@MaksYermak
Copy link
Contributor

In this PR I have added new operators for creating and managing Ray clusters on Vertex AI infrastructure.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Mar 4, 2025
@MaksYermak MaksYermak force-pushed the ray-on-vertex-ai-cluster branch 2 times, most recently from 3b2f7f7 to 2e5a9a9 Compare March 5, 2025 14:41
@VladaZakharova
Copy link
Contributor

Hi @potiuk !
Can you please help us here? We have some import errors, not sure why exactly 😺

@potiuk potiuk force-pushed the ray-on-vertex-ai-cluster branch from 2e5a9a9 to 6184b9e Compare March 6, 2025 18:46
@potiuk
Copy link
Member

potiuk commented Mar 6, 2025

Rebased it - to check - we had some teething issue with devel-common - maybe related.

@potiuk potiuk closed this Mar 6, 2025
@potiuk potiuk reopened this Mar 6, 2025
@MaksYermak MaksYermak force-pushed the ray-on-vertex-ai-cluster branch 2 times, most recently from 9926296 to 3e447b4 Compare March 13, 2025 17:00
@MaksYermak MaksYermak force-pushed the ray-on-vertex-ai-cluster branch from 3e447b4 to f66a983 Compare March 14, 2025 13:49
@MaksYermak
Copy link
Contributor Author

Hello @potiuk !
I have been digging deeper in this import issue with the ray package in this CI. And I have noticed that started from Python Version 3.12 installation extra packages for dependencies using this pip command pip install google-cloud-aiplatform[evaluation,ray] is not worked correctly. These two packages evaluation and ray are not installed. You can reproduce it locally in a pure virtual environment with Python 3.12.

I didn't find any python issues related to this. Do you have any idea what is a better way for Airflow to install extra packages for dependencies for Pyhton>=3.12?

@VladaZakharova
Copy link
Contributor

Hi @potiuk :)
Can you please check this one? I think it influences other imports so we need to find a way to correctly handle that. We also have other upcoming PRs related to Ray, so the solution here will apply for other changes too :)

@potiuk
Copy link
Member

potiuk commented Apr 9, 2025

I missed that one - but I will come back next week.

@MaksYermak MaksYermak force-pushed the ray-on-vertex-ai-cluster branch from f66a983 to b7fb631 Compare April 14, 2025 16:16
@potiuk
Copy link
Member

potiuk commented Apr 14, 2025

Some docs failure . I think it's because path to exampleinclude should be changed - according to https://github.com/apache/airflow/blob/main/contributing-docs/11_documentation_building.rst#including-examples-and-other-files

@MaksYermak MaksYermak force-pushed the ray-on-vertex-ai-cluster branch from b7fb631 to 3562dfb Compare April 24, 2025 13:33
@potiuk potiuk merged commit 7f6340a into apache:main Apr 24, 2025
70 checks passed
prabhusneha pushed a commit to astronomer/airflow that referenced this pull request Apr 25, 2025
jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants