AirbyteLib: Add basic secrets management

[aaronsteers]

This adds a new get_secret() option which retrieves from the following sources, in priority order:

1. Env variables
2. Google Colab (if applicable)
3. User prompt (masked)

This approach allows the same code to run identically, even if the secrets are provided in different ways. For instance, in Colab, a secret might be automatically detected and imported, whereas running locally, the same code will import from an environment variable or otherwise will prompt a user in a masked prompt.

Example from the new run_github.py example script:

import airbyte_lib as ab


GITHUB_TOKEN = ab.get_secret("GITHUB_PERSONAL_ACCESS_TOKEN")


source = ab.get_connector("source-github")
source.set_config({
    "repositories": ["airbytehq/airbyte"],
    "credentials": {
        "personal_access_token": GITHUB_TOKEN
    }
})
source.check()

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 6, 2024 8:29am

[flash1293]

The functionality looks good to me, but the "prompt" source for secrets is incompatible with the live view - like with the normal system prompt, the message to enter the secret flashes shortly, then disappears.

#34782 helps as long as you are in a REPL, but in case you run a python script via python my_script.py, the problem still persists.

Do you plan to implement the auto-loading functionality as described in the tech spec in a follow-up PR?

[aaronsteers]

@flash1293 - Thanks for this review.

Re:

Do you plan to implement the auto-loading functionality as described in the tech spec in a follow-up PR?

Yes. This increment was previously unplanned, but added based on some observations of usage from early testers, and can be incrementally expanded to auto-pull into a setting 'slot' based on name convention.