Gcp quota optimization

[shethhriday29]

Description:

• Drastically reduce failover time when we attempt to provision accelerators in an GCP region where the region's quota for those accelerators is zero
• This involved directly querying every region's accelerator quota immediately before provision, and if the query returns a 0 value, skipping the provision attempt (and potentially moving onto the next region as a result)
• Used GCP's CLI commands for query
• Reduces the failover in these zero-quota cases from 30 seconds (failed provisioning attempt) to <1 second (time for CLI command to run)
• Had to add a file to the catalog (skypilot-catalog, me/@shethhriday29 opened a corresponding PR there) to map accelerators to their quota keywords, which are needed in the CLI commands

Tested (run the relevant ones):

• Created new smoke test, test_gcp_zero_quota_failover, that ensures that the zero quota checker works
• Manually tested functionality by attempting to (across multiple types of accelerators):

1. Provision resources in regions with zero quota, and ensuring that the automatic failover happens
2. Provision resources in non-zero but still not-adequate quota, and ensuring that the provision is still attempted
3. Provision resources in a region with adequate quota and ensuring the provision is successful

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[romilbhardwaj]

Thanks @shethhriday29! I'm thinking of refactoring check_quota_available() a bit. What do you think?

[romilbhardwaj]

Awesome work @shethhriday29! Took a look at the code. Haven't tested yet.

[romilbhardwaj]

Awesome work @shethhriday29! Works nicely. Found a bug in smoke tests (it was passing silently) - should be good to go after that's fixed.

[romilbhardwaj]

Awesome work @shethhriday29! This will save a ton of time during failover on GCP 🚀