Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use S3 inventory to simplify the batching process #18

Closed
austinbyers opened this issue Jul 28, 2017 · 1 comment · Fixed by #131
Closed

Use S3 inventory to simplify the batching process #18

austinbyers opened this issue Jul 28, 2017 · 1 comment · Fixed by #131

Comments

@austinbyers
Copy link
Collaborator

Instead of iterating over the entire S3 bucket, you can configure an S3 inventory which saves copies of the bucket contents.

Unfortunately, this is not yet supported by Terraform: hashicorp/terraform-provider-aws#532

This might still be worth implementing manually to avoid the need for a batching function

@austinbyers
Copy link
Collaborator Author

Closing as "wontfix" for the following reasons:

  • No terraform support
  • It could still take 5+ minutes to enqueue millions of items from inventory into SQS, so having a batcher Lambda would likely still be necessary
  • Inventory could be delayed by a day whereas the current implementation guarantees that your retroactive analysis is valid as of the time the command is executed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant