New pipeline: deregister old AMIs #1414
Conversation
yob
force-pushed
the
clean-old-amis
branch
2 times, most recently
from
3a9f5a3 to
85d4a4e
Compare
We've been publishing AMIs to 20 regions for ~6 years, and we've built quite a collection. Many of them we want to keep indefinitely, like any that we've published in a released Cloud Formation template on [1]. Or any that were published in the last 12 months. However, there's many many other images that it's safe to garbage collect. For example, we publish AMIs to every region for every main branch build, so we and customers can test them. They're useful for a short time, but any older than 12 months we can remove. By that time, customers have the option to move to a formal release with the features they want. [1] https://github.com/buildkite/elastic-ci-stack-for-aws/releases
Start very conservative while we build confidence in this process. We can increase the cap in the future.
| matrix: | ||
| - "us-east-1" | ||
| - "us-west-2" | ||
| - "ap-southeast-2" |
There was a problem hiding this comment.
I'll add the other 17 regions in a follow up PR
| agents: | ||
| queue: "oss-deploy" | ||
| env: | ||
| DRY_RUN: true |
There was a problem hiding this comment.
for now, we're only dry running it. We'll build confidence the output looks like we expect, then remove this
| end | ||
|
|
||
| one_year_ago = Time.now - (60 * 60 * 24 * 365) | ||
| deleted_counter = 0 |
| exit 1 | ||
| end | ||
|
|
||
| MAX_DELETIONS = 10 |
There was a problem hiding this comment.
Should this be an arg/env var? Wondering if this script would be useful to run in anger
There was a problem hiding this comment.
I was thinking of starting with 10, just to contain the blast radius of any early mistakes. Then increasing to something like 100 and letting it make progress via a scheduled build (daily? weekly?)
| image.name.start_with?("buildkite-stack-") || # The name we used until mid 2019 | ||
| image.name.start_with?("buildkite-stack-linux-") || # The name we used for linux amd64/arm64 from mid 2019 | ||
| image.name.start_with?("buildkite-stack-windows-") # The name we used for windows amd64 from mid 2019 | ||
| } |
There was a problem hiding this comment.
I would prefer if we had a tag on the AMIs that we could use to find 100% of elastic stack images. Maybe ElasticStack: true or similar. This is a good start for now though
We've been publishing AMIs to 20 regions for ~6 years, and we've built quite a collection.
Many of them we want to keep indefinitely, like any that we've published in a released Cloud Formation template on [1]. Or any that were published in the last 12 months.
However, there's many many other images that it's safe to garbage collect. For example, we publish AMIs to every region for every main branch build, so we and customers can test them. They're useful for a short time, but any older than 12 months we can remove. By that time, customers have the option to move to a formal release with the features they want.
[1] https://github.com/buildkite/elastic-ci-stack-for-aws/releases