Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata Upload: target production GCS bucket #25529

Merged
merged 5 commits into from
Apr 26, 2023

Conversation

bnchrch
Copy link
Contributor

@bnchrch bnchrch commented Apr 25, 2023

What

We need metadata files uploading to the production bucket
closes #25091

How

This updates publish to target prod-airbyte-cloud-connector-metadata-service instead of dev-airbyte-cloud-connector-metadata-service

This also updates our spec mask to generate at the same time as the registry

Before merge

  • Add production service account to Dagster
  • Update dagster metadata bucket env var to dev-airbyte-cloud-connector-metadata-service

After merge

  • Copy metadata folder from dev bucket to prod
  • Trigger a manual generation of assets
  • Enable sensors

@bnchrch bnchrch requested review from alafanechere and a team April 25, 2023 23:26
@bnchrch bnchrch changed the title Metadata: target production GCS bucket Metadata Upload: target production GCS bucket Apr 25, 2023
Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat!
Do you think we should double check that the current branch is master, and use the dev bucket when it's not?

@@ -28,7 +28,7 @@ jobs:
GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
GCS_CREDENTIALS: ${{ secrets.METADATA_SERVICE_DEV_GCS_CREDENTIALS }}
METADATA_SERVICE_ACCOUNT_KEY: ${{ secrets.METADATA_SERVICE_DEV_GCS_CREDENTIALS }}
Copy link
Contributor

@alafanechere alafanechere Apr 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bnchrch this SA should be updated too. I think we now have permission to create sercrets.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

@bnchrch bnchrch force-pushed the bnchrch/dagster/target-production-bucket branch from 18363dd to 830c721 Compare April 26, 2023 19:35
@bnchrch bnchrch requested a review from alafanechere April 26, 2023 19:35
@bnchrch bnchrch merged commit bd43a4f into master Apr 26, 2023
@bnchrch bnchrch deleted the bnchrch/dagster/target-production-bucket branch April 26, 2023 23:43
marcosmarxm pushed a commit to natalia-miinto/airbyte that referenced this pull request Jun 8, 2023
* Update spec mask to be generated from sensor

* Update publish to use production bucket

* Update env variable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update dagster cloud and metadata-service to reference production bucket
2 participants