-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Elastic JobSets #463
Comments
Yes, I think we should think about allowing to autoscale the number of replicas in a replicatedJob! For example, the number of tpuslices (or more generally accelerator islands) supporting a large scale training job could scale down in case of failures.
The jobs are indexed, so a scale down means removing the higher order ones.
It is possible. If the child jobs themselves should be elastic, then the operator could change the individual jobs directly. But I guess we could also allow changing that in bulk for all job replicas, but I need to hear a use case first. |
So @ahg-g it sounds like this is supported as you are correct. Both replicas in ReplicatedJob are mutable and JobTemplate is mutable.. Maybe we should consider a task for this to at least document that this is possible? I think Kueue or other use cases would be interested in this but not sure what we need in this repo. |
We need to have tests for that to verify the behavior though. |
Well good thing I tried it haha. I think this code is blocking us from doing this. I tried a simple example and did a Got:
|
/retitle Support Elastic JobSets |
I opened up #465 for discussion. We were treating the entire replicated job as immutable. It isn't clear to me what validation logic we want to have for a replicated job. We could go with just changing replicas (name and JobTemplate are immutable). |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
@andreyvelich @tenzen-y Could you comment on what kind of behavior you'd expect from JobSet for Elastic PyTorch? |
What would you like to be added:
With Elastic Indexed jobs, it is possible to change completions/parallelism to down/up scale your jobs.
It would be nice to have something similar for JobSet.
Why is this needed:
Elastic jobs are an important usecase for autoscaling and other cases.
Implementation:
At a quick glance of the API this may be possible as replicas of a replicated job are not immutable so I think someone could patch the replicas of a ReplicatedJob to downscale or upscale.
But then I wonder what should we do with the existing replicated job?
And should we support ElasticIndexedJob with JobSet (so someone could patch the JobTemplate in a single replicated job?
The text was updated successfully, but these errors were encountered: