Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CustomContainerTrainingJob does not expose timeout parameter #1032

Closed
jasonbrancazio opened this issue Feb 22, 2022 · 3 comments
Closed

CustomContainerTrainingJob does not expose timeout parameter #1032

jasonbrancazio opened this issue Feb 22, 2022 · 3 comments
Assignees
@ivanmkc
Labels
aiplatform Issues related to the AI Platform (Unified) API. api: aiplatform Issues related to the AI Platform API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@jasonbrancazio
Copy link

The CustomJobSpec for a Vertex AI contains a Scheduling object.

The Scheduling object includes a timeout field that allows a user to specify a maximum job running time.

Unfortunately, the run method of the CustomContainerTrainingJob
does not expose the timeout field.

The lack of this feature exposes a user of Vertex AI pipelines to the risk of unintended costs from long-running jobs.
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Feb 23, 2022
@busunkim96 busunkim96 added aiplatform Issues related to the AI Platform (Unified) API. api: aiplatform Issues related to the AI Platform API. labels Feb 23, 2022
@ivanmkc
Copy link
Contributor

ivanmkc commented Feb 23, 2022

Good catch, this feature was actually added recently and will be available in the next release (early March).

PR: #970

@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Feb 27, 2022
@kweinmeister kweinmeister added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Mar 3, 2022
@morgandu
Copy link
Contributor

@sararob , can you please verify the timeout PR covers this issue as well.

@sasha-gitg
Copy link
Member

This has been added: https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/training_jobs.py#L2673

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ivanmkc ivanmkc

Labels
aiplatform Issues related to the AI Platform (Unified) API. api: aiplatform Issues related to the AI Platform API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@ivanmkc @sararob @busunkim96 @morgandu @kweinmeister @jasonbrancazio @sasha-gitg @yoshi-automation