Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add jobs that trigger on file upload #527

Merged
merged 6 commits into from
Feb 6, 2024
Merged

Add jobs that trigger on file upload #527

merged 6 commits into from
Feb 6, 2024

Conversation

lorenyu
Copy link
Contributor

@lorenyu lorenyu commented Dec 21, 2023

Ticket

Resolves #522

Changes

  • Refactor service name and bucket name to env-config module
  • Add file_upload_jobs config to env-config module
  • Update service module to add event rule and target to trigger jobs based on file uploads
  • Add role and policy that EventBridge will use to run ECS tasks
  • Document background job functionality

Example app changes

  • Add etl command to example Flask app
  • Add etl job to file_upload_jobs config for platform-test

CI changes

  • Bump terraform version in ci-infra-service workflow

Fixes

  • Fixed storage access permissions to have kms:Decrypt

Unrelated changes

  • Moved feature flags config to separate file in same module

Context for reviewers

Many applications need a basic ETL system that ingests files that are uploaded to a bucket. This change adds that functionality.

Testing

Developed and tested on platform-test in navapbc/platform-test#81

Ignore the anchor scan failure since it's in the example app and I don't want to add an ignore rule to the template anchor scan config.

Copy link
Contributor

@daphnegold daphnegold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looked good to me except where I got very stuck trying to understand what's going on with the input_transformer. 🥇

docs/infra/background-jobs.md Outdated Show resolved Hide resolved
infra/modules/service/jobs.tf Show resolved Hide resolved
Co-authored-by: Daphne Gold <daphnegold@navapbc.com>
@charles-nava charles-nava self-requested a review February 6, 2024 17:35
Copy link
Contributor

@charles-nava charles-nava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@lorenyu lorenyu merged commit 164e7e4 into main Feb 6, 2024
13 of 14 checks passed
@lorenyu lorenyu deleted the lorenyu/events branch February 6, 2024 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Jobs triggered by file upload
3 participants