-
Notifications
You must be signed in to change notification settings - Fork 360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GCP Batch: Support passing standard machine types to the Google backend #7545
base: develop
Are you sure you want to change the base?
GCP Batch: Support passing standard machine types to the Google backend #7545
Conversation
…ng the cpuPlatform field in the RuntimeAttributes
...gle/batch/src/main/scala/cromwell/backend/google/batch/util/GcpBatchMachineConstraints.scala
Outdated
Show resolved
Hide resolved
366115d
to
4ba50c3
Compare
...ogle/batch/src/main/scala/cromwell/backend/google/batch/api/GcpBatchRequestFactoryImpl.scala
Show resolved
Hide resolved
…using cpuPlatform
4ba50c3
to
3a0ac16
Compare
@aednichols @mcovarr : Please let us know if you have any feedback on this PR. I would like to confirm this is in the right direction before working on tests and getting this ready for merging. Alternatively, please let me know if I should reach out to someone else for review. |
Thanks for taking the time to submit this! Unfortunately the maintaining team doesn't have much bandwidth to collaborate on community contributions right now, but we'll keep it in mind for the future. |
…achine-type-google-backend
Description
Currently, and as described in #7535, only general-purpose machine types are supported in Google Backend, which prevents running wdl workflows on many machine types available on GCP, including those provisioned with modern GPUs.
I believe the simplest and most general solution would be to pass the machine type directly from the wdl configuration to the Google Batch API. The idea is that this approach would be more resilient to machine types being added or deprecated on GCP, as users would only need to update their wdl workflows in such cases. An alternative approach of mapping machine specs (e.g.: cpu platform and gpu requirements) to standard machine types would potentially introduce an additional layer of maintenance with little benefit.
This PR adds support for a new standardMachineType key in the runtime section, which is only parsed for the Google backend.
Testing
I deployed this internally and verified I can successfully run the following wdl workflow:
Next steps
Release Notes Confirmation
CHANGELOG.md
CHANGELOG.md
in this PRCHANGELOG.md
because it doesn't impact community usersTerra Release Notes