Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ETL-365] Launch stacks during push and PR to dev AWS account #24

Merged
merged 8 commits into from
Mar 16, 2023
Merged

Conversation

rxu17
Copy link
Contributor

@rxu17 rxu17 commented Mar 16, 2023

Purpose: This PR follows the same logic as BridgeDownstream in that it launches stacks to the "recover" and feature branch namespaces of the recover develop account.

This PR also parameterizes some of the python and glue versions used in the glue job stacks as we have varying python and glue versions to be used in the highest level stack config:config/config.yaml. Was unable to dynamically parameterize the lambda python runtime in the sam template to this stack because findings say it's not possible to pass this through to the lambda sam template.

  • JSON to Parquet glue job: Glue version 4.0 which supports Python 3.10
  • S3 to Parquet glue job: Python 3.9 (this is their highest available python for the glue python shell jobs) which means it can only use Glue version 3.0
  • Lambda function for S3 to JSON: Python 3.9 (this is their highest python version that they support, but 3.10 should be coming along in a couple of months)
  • Pipenv python/overall environment: Python 3.9 otherwise we can't sam build lambda especially when using GH actions

@rxu17 rxu17 requested a review from a team as a code owner March 16, 2023 20:11
Copy link
Member

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👽 LGTM! This is a lot of combinations of dependencies.. Hopefully someday the dependencies are all supported by AWS...

@rxu17 rxu17 merged commit 008d0e9 into main Mar 16, 2023
@rxu17 rxu17 deleted the etl-365 branch March 16, 2023 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants