ci-credentials: update GSM secrets with updated configuration values

[alafanechere]

What

Closes #19441 (2/2)

#19979 enabled SAT to write updated configurations by capturing CONTROL messages.
When running in the CI, we need to upload these updated configurations to GSM. This is the scope of this PR.

How

1. Refactor ci-credentials to expose two commands via click framework:

• write-to-storage: the already existing command that downloads GSM secrets locally for test runs
• update-secrets: the new command to update GSM secrets according to the content of the secrets/updated_configurations folder that might contain SAT updated configs. This command finds the latest local updated configuration and updates the existing GSM secret by creating a new secret version and disabling the previous one.
  

  airbyte/tools/ci_credentials/ci_credentials/main.py

  Line 23 in 1e75110

  def ci_credentials(ctx, connector_name: str, gcp_gsm_credentials):

2. Create Secret and RemoteSecret models for convenient manipulation of secret structure.
   

   airbyte/tools/ci_credentials/ci_credentials/models.py

   Line 15 in 1e75110

   class Secret:

3. Rename SecretLoader to SecretManager and add methods to update secrets and disable version.

airbyte/tools/ci_credentials/ci_credentials/secrets_manager.py

Line 246 in 1e75110

def update_secrets(self, existing_secrets: List[RemoteSecret]) -> int:

4. Test SecretManager public methods and models:

airbyte/tools/ci_credentials/tests/test_secrets_manager.py

Line 57 in 1e75110

def test_read(matchers, connector_name, gsm_secrets, expected_secrets):

airbyte/tools/ci_credentials/tests/test_models.py

Line 44 in 1e75110

def test_secret_instantiation(connector_name, filename, expected_name, expected_directory):

5. Change Github workflows using ci-credentials to make them call the new update-secrets commands after test run.

🚨 User Impact 🚨

This PR changes the /test and /publish workflows, so we shall make sure this command are still functioning as expected.

[alafanechere]

I manually dispatched a test here on source-facebook-marketing to check that the updated test-command workflow is still working. I could not use /test command dispatch as it checkouts the workflow from master.

Update: the test-command workflow ran successfully 🎉 so the test run on existing connector (not emitting CONTROL message) are likely to not be disturbed by my changes.

[evantahler]

https://click.palletsprojects.com/en/8.1.x/ is neat!

[evantahler]

Awesome work! Just one main question below, while waiting on others who know more about 🐍

[evantahler]

Confirming that GCP_GSM_CREDENTIALS has read and write access?

[alafanechere]

I'm not allowed to check in GitHub which service account is used in this env var in our actions. But according to the service account I see in our IAM console, I'm pretty sure the currently used SA has sufficient permissions to perform all the operations that this CLI can do: read secret values, add a secret version, disable a secret version.

[sherifnada]

Props for refactoring and adding tests. This is great for maintainability

[alafanechere]

https://click.palletsprojects.com/en/8.1.x/ is neat!

Yup, it currently powers octavia-cli and cloud-cli.