ci-automation/garbage_collect.sh: Add min age, remove orphan directories

[t-lo]

This change improves the build cache garbage collector to remove orphaned artifact directories - i.e. directories to which no version tag exists in the scripts repo.

SDK containers built by Github actions (using update_sdk_container) are igored by this change because these are handled in a separate garbage collection script.

Also, a new command line parameter has been added to remove artifacts older than the specified number of days (defaulting to 14):

• If neither number of builds nor max age is specified, the script defaults to 50 builds to keep, and a max age of 14 days. The max age overrides the number of builds to keep, so more than 50 builds may be kept.
• If only the number of builds to keep is specified, the max age is set to "0" (i.e. today).
• If both are specified, max age again overrides number of builds to keep.

Lastly, the change updates ci-automation/garbage_collect_github_ci_sdk.sh to also feature a min_age parameter and passes the parameter from ``garbage_collect.shtoci-automation/garbage_collect_github_ci_sdk.sh`.

How to use

bash -c "source ci-automation/garbage_collect.sh ; DRY_RUN=y garbage_collect ; "
bash -c "source ci-automation/garbage_collect.sh ; DRY_RUN=y garbage_collect 45; "
bash -c "source ci-automation/garbage_collect.sh ; DRY_RUN=y garbage_collect 1 12; "

bash -c "source ci-automation/garbage_collect_github_ci_sdk.sh ; DRY_RUN=y garbage_collect_github_ci 1 236; "
bash -c "source ci-automation/garbage_collect_github_ci_sdk.sh ; DRY_RUN=y garbage_collect_github_ci 10 200; "

Testing done

Ran the above 5 commands and attached logs:

• remove orphans - remove_orphans_run.txt (GH clean-up removed from log for readability)
• remove all but last 45 builds (orphan purging and GH clean-up removed from log for readability) - remove_all_but_45_run.txt
• remove all but 1 builds but minimum age set to 12 days (orphan purging and GH clean-up removed from log for readability) - remove_all_older_than_12_days_run.txt
• Github actions cleanup of artifacts older than 236 days - remove_gh_ci_sdk_older_236_run.txt
• Github actions cleanup of artifacts older than 200 days, keep at least 10 - remove_gh_ci_sdk_older_200_keep_10_run.txt

Follow-up tasks

• Integrate max age parameter with jenkins-os

[krnowak]

Just one SDK build to keep? Asking to make sure, especially that the default was 20. I do realize that probably more will be kept until they are older than 14 days or so.

[t-lo]

The "1" just means that at least one is kept; for everything else the 14-day limit will apply. We don't build the SDK from GitHub actions that often so I thought that would be sensible.

[krnowak]

Why excluding the -pro build tag? Some artifact from the past?

[t-lo]

Yes, to prevent deleting the release tags of the -pro versions.

[krnowak]

This is a bit too all-catching as it will prevent GC of any dev build that ends with -pro (so a build tag like weekly-updates-like-a-pro won't be GC'd), but OTOH we tend not to put -pro into our build tags, so I suppose it's fine.

[t-lo]

Good point, the match could be more precise.

[krnowak]

Same here for excluding -pro from purges.

[t-lo]

We would need to rework tag removal and exclude any pattern that matches an official release version. Note that versions that look like official releases are skipped entirely - the garbage collector only cares about dev builds and nightlies. (Releases are supposed to be cleaned up manually, after pushing release binaries to the official mirrors.)