Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move the prune-uploads script to AWS Batch #2216

Closed
iamleeg opened this issue Nov 1, 2021 · 1 comment
Closed

Move the prune-uploads script to AWS Batch #2216

iamleeg opened this issue Nov 1, 2021 · 1 comment
Assignees
@abhidg
Labels
Backlog Importer Infrastructure Issues related to AWS/EKS/Docker etc P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix
Milestone
Leonidas

Comments

@iamleeg
Copy link
Contributor

iamleeg commented Nov 1, 2021

It currently takes 6h to run the prune script for the US data set which is right on the runtime limit for GitHub actions.

While here, we also need better sequencing via hooks or queueing for ingestion, pruning, exporting, and aggregation to ensure that downloads and reports are consistent with a stable view of the data.
@iamleeg iamleeg added this to the Holding Bin milestone Nov 1, 2021
@iamleeg iamleeg added Importer Infrastructure Issues related to AWS/EKS/Docker etc labels Nov 1, 2021
@abhidg abhidg self-assigned this Nov 1, 2021
@iamleeg iamleeg modified the milestones: Holding Bin, Lewis Nov 3, 2021
@joe-brilliant joe-brilliant added the P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix label Dec 2, 2021
@jim-sheldon jim-sheldon mentioned this issue Dec 16, 2021
Merged
@joe-brilliant joe-brilliant added P2: Nice to have This would be nice to have, if we have time we will fix it, if not good to launch and fix later when and removed P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix labels Jan 13, 2022
@joe-brilliant joe-brilliant modified the milestones: Lewis, Holding Bin Feb 1, 2022
@abhidg abhidg modified the milestones: Holding Bin, Leonidas Feb 21, 2022
@abhidg abhidg added P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix and removed P2: Nice to have This would be nice to have, if we have time we will fix it, if not good to launch and fix later when labels Feb 21, 2022
@abhidg
Copy link
Contributor

abhidg commented Feb 21, 2022

Bumping to P1 as non-US ingestions are timing out as well

abhidg added a commit that referenced this issue Feb 21, 2022
@abhidg
prune: move to Batch #2216
b1cc58a
abhidg added a commit that referenced this issue Feb 21, 2022
@abhidg
prune: move to Batch #2216
60276f9
abhidg added a commit that referenced this issue Feb 21, 2022
@abhidg
prune: move to Batch #2216
bb8ee2e
abhidg added a commit that referenced this issue Feb 21, 2022
@abhidg
prune: move to Batch #2216
da28a11
abhidg added a commit that referenced this issue Feb 21, 2022
@abhidg
prune: move to Batch #2216
92be581
abhidg added a commit that referenced this issue Feb 22, 2022
@abhidg
prune: move to Batch #2216
b91a488
@abhidg abhidg closed this as completed Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@abhidg abhidg

Labels
Backlog Importer Infrastructure Issues related to AWS/EKS/Docker etc P1: Launch blocker Needs fixing before we launch, schedule some time to investigate & fix
Projects
None yet
Milestone
Leonidas
Development

No branches or pull requests
3 participants
@abhidg @iamleeg @joe-brilliant