GCP Batch: Support passing standard machine types to the Google backend #7545
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently, and as described in #7535, only general-purpose machine types are supported in Google Backend, which prevents running wdl workflows on many machine types available on GCP, including those provisioned with modern GPUs.
I believe the simplest and most general solution would be to pass the machine type directly from the wdl configuration to the Google Batch API. The idea is that this approach would be more resilient to machine types being added or deprecated on GCP, as users would only need to update their wdl workflows in such cases. An alternative approach of mapping machine specs (e.g.: cpu platform and gpu requirements) to standard machine types would potentially introduce an additional layer of maintenance with little benefit.
This PR adds support for a new standardMachineType key in the runtime section, which is only parsed for the Google backend.
Testing
I deployed this internally and verified I can successfully run the following wdl workflow:
Next steps
Release Notes Confirmation
CHANGELOG.md
CHANGELOG.md
in this PRCHANGELOG.md
because it doesn't impact community usersTerra Release Notes