-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci-automation/garbage_collect.sh: Add min age, remove orphan directories #1608
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,10 +8,29 @@ | |
# | ||
# garbage_collect() should be called after sourcing. | ||
# | ||
# The garbage collector will remove artifacts of all NON-RELEASE versions from the build cache | ||
# which BOTH | ||
# * exceed the number of builds to keep (defaults to 50) | ||
# AND | ||
# * are older than the minimum purge age (14 days by default) | ||
# | ||
# Note that the min age threshold can lead to MORE than 50 builds being kept if this script | ||
# is run with its default values. | ||
# | ||
# Additionally, the garbage collector will remove all artifacts and directories that do not have | ||
# a version TAG in the scripts repository. | ||
# | ||
# OPTIONAL INPUT | ||
# - Number of (recent) versions to keep. Defaults to 50. | ||
# Explicitly setting this value will reset the minimum age (see below) to 0 days. | ||
# - Minimum age of version tag to be purged, in days. Defaults to 14. | ||
# Only artifacts older than this AND exceeding the builds to keep threshold | ||
# will be removed. | ||
# - PURGE_VERSIONS (Env variable). Space-separated list of versions to purge | ||
# instead of all but the 50 most recent ones. | ||
# Setting this will IGNORE minimum age and number of versions to keep. | ||
# NOTE that only dev versions (not official releases) may be specified. | ||
# This is to prevent accidental deletion of official release tags from the git repo. | ||
# - DRY_RUN (Env variable). Set to "y" to just list what would be done but not | ||
# actually purge anything. | ||
|
||
|
@@ -38,26 +57,60 @@ function garbage_collect() { | |
# -- | ||
|
||
function _garbage_collect_impl() { | ||
local keep="${1:-50}" | ||
local keep="${1:-}" | ||
local min_age_days="${2:-}" | ||
local dry_run="${DRY_RUN:-}" | ||
local purge_versions="${PURGE_VERSIONS:-}" | ||
|
||
local versions_detected="$(git tag -l --sort=-committerdate \ | ||
| grep -E '(main|alpha|beta|stable|lts)-[0-9]+\.[0-9]+\.[0-9]+\-.*' \ | ||
| grep -vE '(-pro)$')" | ||
# Set defaults; user-provided 'keep' has priority over default 'min_age_days' | ||
if [ -n "${keep}" -a -z "${min_age_days}" ] ; then | ||
min_age_days="0" | ||
elif [ -z "${keep}" ] ; then | ||
keep="50" | ||
fi | ||
if [ -z "${min_age_days}" ] ; then | ||
min_age_days="14" | ||
fi | ||
|
||
echo "######## Full list of version(s) found ########" | ||
echo "${versions_detected}" | awk '{printf "%5d %s\n", NR, $0}' | ||
local min_age_date="$(date -d "${min_age_days} days ago" +'%Y-%m-%d')" | ||
echo "######## Garbage collector starting ########" | ||
echo | ||
if [ -z "${purge_versions}" ] ; then | ||
echo "Number of versions to keep: '${keep}'" | ||
echo "Keep newer than: '${min_age_date}' (overrides number of versions to keep)" | ||
fi | ||
echo | ||
|
||
if [ -z "${purge_versions}" ] ; then | ||
keep="$((keep + 1))" # for tail -n+... | ||
# Generate a list "<timestamp> | <tagname>" from all repo tags that look like dev versions | ||
local versions_detected="$(git tag -l --sort=-committerdate \ | ||
--format="%(creatordate:format:%Y-%m-%d) | %(refname:strip=2)" \ | ||
| grep -E '.*\| (main|alpha|beta|stable|lts)-[0-9]+\.[0-9]+\.[0-9]+-.*' \ | ||
| grep -vE '(-pro)$')" | ||
|
||
echo "######## Full list of version(s) and their creation dates ########" | ||
echo | ||
echo "${versions_detected}" | awk '{printf "%5d %s\n", NR, $0}' | ||
|
||
# Filter minimum number of versions to keep, min age | ||
purge_versions="$(echo "${versions_detected}" \ | ||
| tail -n+"${keep}")" | ||
| awk -v keep="${keep}" -v min_age="${min_age_date}" '{ | ||
if (keep > 0) { | ||
keep = keep - 1 | ||
next | ||
} | ||
|
||
if ($1 > min_age) | ||
next | ||
|
||
print $3 | ||
}')" | ||
else | ||
# make sure we only accept dev versions | ||
# User-provided version list, make sure we only accept dev versions | ||
purge_versions="$(echo "${purge_versions}" | sed 's/ /\n/g' \ | ||
| grep -E '(main|alpha|beta|stable|lts)-[0-9]+\.[0-9]+\.[0-9]+\-.*' \ | ||
| grep -vE '(-pro)$')" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here for excluding There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We would need to rework tag removal and exclude any pattern that matches an official release version. Note that versions that look like official releases are skipped entirely - the garbage collector only cares about dev builds and nightlies. (Releases are supposed to be cleaned up manually, after pushing release binaries to the official mirrors.) |
||
keep=0 | ||
fi | ||
|
||
source ci-automation/ci_automation_common.sh | ||
|
@@ -71,7 +124,7 @@ function _garbage_collect_impl() { | |
echo "(NOTE this is just a dry run since DRY_RUN=y)" | ||
echo | ||
fi | ||
echo "${purge_versions}" | awk -v keep="${keep}" '{if ($0 == "") next; printf "%5d %s\n", NR + keep - 1, $0}' | ||
echo "${purge_versions}" | awk '{if ($0 == "") next; printf "%5d %s\n", NR, $0}' | ||
echo | ||
echo | ||
|
||
|
@@ -90,7 +143,7 @@ function _garbage_collect_impl() { | |
local os_docker_vernum="$(vernum_to_docker_image_version "${FLATCAR_VERSION}")" | ||
|
||
# Remove container image tarballs and SDK tarball (if applicable) | ||
# | ||
# Keep in sync with "orphaned directories" clean-up below. | ||
local rmpat="" | ||
rmpat="${BUILDCACHE_PATH_PREFIX}/sdk/*/${os_vernum}/" | ||
rmpat="${rmpat} ${BUILDCACHE_PATH_PREFIX}/containers/${os_docker_vernum}/flatcar-sdk-*" | ||
|
@@ -144,6 +197,54 @@ function _garbage_collect_impl() { | |
fi | ||
done | ||
|
||
echo | ||
echo "########################################" | ||
echo | ||
echo Checking for orphaned directories | ||
echo | ||
|
||
local dir="" | ||
for dir in "sdk/amd64" \ | ||
"containers" \ | ||
"boards/amd64-usr" \ | ||
"boards/arm64-usr" \ | ||
"images/amd64" \ | ||
"images/arm64" \ | ||
"testing" \ | ||
; do | ||
local fullpath="${BUILDCACHE_PATH_PREFIX}/${dir}" | ||
echo | ||
echo "## Processing '${fullpath}'" | ||
echo "---------------------------" | ||
local version="" | ||
for version in $($sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" "ls -1 ${BUILDCACHE_PATH_PREFIX}/${dir}"); do | ||
if [ "${dir}" = "containers" ] && echo "${version/+/-}" | grep -qE '.*-github-.*'; then | ||
echo "Ignoring github CI SDK container in '${fullpath}/${version}'." | ||
echo "Github CI SDK artifacts are handled by 'garbage_collect_github_ci_sdk.sh'" | ||
echo " in a later step". | ||
continue | ||
fi | ||
if ! git tag -l | grep -q "${version/+/-}"; then | ||
local o_fullpath="${fullpath}/${version}" | ||
echo | ||
echo "## No tag '${version/+/-}' for orphan directory '${o_fullpath}'; removing." | ||
echo "## The following files will be removed ##" | ||
$sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" \ | ||
"ls -la ${o_fullpath} || true" | ||
|
||
if [ "$dry_run" != "y" ] ; then | ||
set -x | ||
$sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" \ | ||
"rm -rf ${o_fullpath} || true" | ||
set +x | ||
else | ||
echo "## (DRY_RUN=y so not doing anything) ##" | ||
fi | ||
echo | ||
fi | ||
done | ||
done | ||
|
||
echo | ||
echo "########################################" | ||
echo | ||
|
@@ -170,6 +271,6 @@ function _garbage_collect_impl() { | |
echo | ||
|
||
source ci-automation/garbage_collect_github_ci_sdk.sh | ||
garbage_collect_github_ci | ||
garbage_collect_github_ci 1 "${min_age_days}" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just one SDK build to keep? Asking to make sure, especially that the default was 20. I do realize that probably more will be kept until they are older than 14 days or so. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The "1" just means that at least one is kept; for everything else the 14-day limit will apply. We don't build the SDK from GitHub actions that often so I thought that would be sensible. |
||
} | ||
# -- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why excluding the
-pro
build tag? Some artifact from the past?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, to prevent deleting the release tags of the
-pro
versions.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bit too all-catching as it will prevent GC of any dev build that ends with -pro (so a build tag like
weekly-updates-like-a-pro
won't be GC'd), but OTOH we tend not to put -pro into our build tags, so I suppose it's fine.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, the match could be more precise.