2217 delete old ADI source data #2506
Conversation
based on the schedule rule for the ingestion source. #2217
|
GitHub Actions has a service account for AWS setup (that's how we deploy to ECR), so if this script takes <6h it can be that, otherwise a Batch job |
Nice, makes sense to be a daily GitHub action I think. |
| def s3_object_key_helper(source_id, date, file_name): | ||
| return f"{source_id}/{date.year}/{date.month}/{date.day}/{file_time}/{file_name}" | ||
|
|
||
|
|
There was a problem hiding this comment.
Should all methods only run in the dockerized environment?
There was a problem hiding this comment.
As this is called by the test and the test only runs in the docker environment, I think that's already true? Nothing is going to run this function otherwise.
ingestion/functions/aws_access/globaldothealth_configuration.py
Outdated
Show resolved
Hide resolved
I would prefer we keep it in AWS, and thus terraformed, as it would keep infrastructure consolidated and might be part of partner requirements. |
Draft because I still need to integrate this into the LocalStack e2e test runner (and into ops too), but asking for feedback on the script and the approach. Also, how will this get run? Is there somewhere I can run this on a schedule that already has access to a service account for our AWS?