cmd-cloud-prune: Prune images for builds

[gursewak1997]

Extend the garbage collection to the images for the builds. We will prune all the images apart from what is specified in the images_keep list for each stream in gc-policy.yaml

[gursewak1997]

One question pertaining to the images pruning is whether we should update the meta.json for each build specifying/deleting the pruned images from the images key?

[dustymabe]

One question pertaining to the images pruning is whether we should update the meta.json for each build specifying/deleting the pruned images from the images key?

I think we should keep them for historical reference. It's also safer to not have to touch other files in s3 (I don't want to get it wrong).

[dustymabe]

Can you open a corresponding PR to the pipeline that proposes updates to the gc-policy.yaml based on these changes? I think it will help to illustrate and drive the review process.

[dustymabe]

Initial round. I've got just a little left to do but will have to pick it up later.

[gursewak1997]

I think we should keep them for historical reference. It's also safer to not have to touch other files in s3 (I don't want to get it wrong).

I thought the same. Ideally, we shouldn't modify meta.json to ensure it's safe and to minimize updates to S3.

[gursewak1997]

Can you open a corresponding PR to the pipeline that proposes updates to the gc-policy.yaml based on these changes? I think it will help to illustrate and drive the review process.

coreos/fedora-coreos-pipeline#1038

[dustymabe]

NOTE: I don't want to make the following suggested change in this PR. If we did decide to do it I'd propose it for a separate PR after this one merges.

Ultimately I think it may be better here to iterate over builds at the highest level for loop rather than the action (policy). i.e.

for build in builds:
    for action in ['cloud-uploads', 'images', 'build']:

rather than how it is now. A few reasons why:

1. right now we download a build's meta.json (for each arch) 3 times (for each action in the policy) from s3. I think that means we download a meta.json 12 times, rather than 4 for each build in the worst case. It's a small file so doesn't matter a ton, but worth noting.
2. This change might allow us to update the build.json file less times with the results of the cleanup.

[dustymabe]

Looking good. One change needed.

[dustymabe]

I think we're missing the key piece:

Suggested change

	print(f"Build {build_id} has already had {action} pruning completed")
	print(f"Build {build_id} has already had {action} pruning completed")
	continue

[dustymabe]

LGTM. Let's merge this and then discuss if we think it's worth a followup for the proposal in #3867 (comment) or not.