Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci_credentials: fix overwriting 'data' before getting nextPageToken #24265

Merged
merged 2 commits into from
Mar 21, 2023
Merged

ci_credentials: fix overwriting 'data' before getting nextPageToken #24265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bbdc4c9
ci_credentials: fix overwriting 'data' before getting nextPageToken
erohmensing Mar 20, 2023
4496032
Updates to readme
erohmensing Mar 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions tools/ci_credentials/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Download a Service account json key that has access to Google Secrets Manager.
* Click on "ADD KEY -> Create new key" and select JSON. This will download a file on your computer

### Setup ci_credentials
* In your .zshrc, add: export GCP_GSM_CREDENTIALS=`cat <path to JSON file>`
* In your .zshrc, add: `export GCP_GSM_CREDENTIALS=cat $(<path to JSON file>)`
* Follow README.md under `tools/ci_credentials`

After making a change, you have to reinstall it to run the bash command: `pip install --quiet -e ./tools/ci_*`
Expand All @@ -44,12 +44,19 @@ The `VERSION=dev` will make it so it knows to use your local current working dir
ci_credentials --help
```

### Write to storage
### Write credentials for a specific connector to local storage
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏

To download GSM secrets to `airbyte-integrations/connectors/source-bings-ads/secrets`:
```bash
ci_credentials source-bing-ads write-to-storage
```

### Write credentials for all connectors to local storage
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏

To download GSM secrets to for all available connectors into their respective `secrets` directories:
```bash
ci_credentials all write-to-storage
```


### Update secrets
To upload to GSM newly updated configurations from `airbyte-integrations/connectors/source-bings-ads/secrets/updated_configurations`:

Expand Down
14 changes: 7 additions & 7 deletions tools/ci_credentials/ci_credentials/secrets_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ def __load_gsm_secrets(self) -> List[RemoteSecret]:
if next_token:
params["pageToken"] = next_token

data = self.api.get(url, params=params)
for secret_info in data.get("secrets") or []:
all_secrets_data = self.api.get(url, params=params)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 Nice rename

for secret_info in all_secrets_data.get("secrets") or []:
secret_name = secret_info["name"]
connector_name = secret_info.get("labels", {}).get("connector")
if not connector_name:
Expand All @@ -103,14 +103,14 @@ def __load_gsm_secrets(self) -> List[RemoteSecret]:
self.logger.info(f"found GSM secret: {log_name} = > {filename}")

versions_url = f"https://secretmanager.googleapis.com/v1/{secret_name}/versions"
data = self.api.get(versions_url)
enabled_versions = [version["name"] for version in data["versions"] if version["state"] == "ENABLED"]
versions_data = self.api.get(versions_url)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Dreaming) If only python had an immutable variable feature

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lol, too spoiled by java's pickiness!

enabled_versions = [version["name"] for version in versions_data["versions"] if version["state"] == "ENABLED"]
if len(enabled_versions) > 1:
self.logger.critical(f"{log_name} should have one enabled version at the same time!!!")
enabled_version = enabled_versions[0]
secret_url = f"https://secretmanager.googleapis.com/v1/{enabled_version}:access"
data = self.api.get(secret_url)
secret_value = data.get("payload", {}).get("data")
secret_data = self.api.get(secret_url)
secret_value = secret_data.get("payload", {}).get("data")
if not secret_value:
self.logger.warning(f"{log_name} has empty value")
continue
Expand All @@ -126,7 +126,7 @@ def __load_gsm_secrets(self) -> List[RemoteSecret]:
remote_secret = RemoteSecret(connector_name, filename, secret_value, enabled_version)
secrets.append(remote_secret)

next_token = data.get("nextPageToken")
next_token = all_secrets_data.get("nextPageToken")
if not next_token:
break

Expand Down