Set up annex OSTree repo

[jlebon]

We should set up an OSTree repo somewhere which holds all the OSTree content from our builds. This is mentioned in the stream tooling doc. This will also be useful for e2e testing (see e.g. #228).

Now, as to where to actually host this repo... the obvious choice is also in the S3 bucket, though it might be painful to manage: coreos/coreos-assembler#159 (comment). The TL;DR is: uploading is OK, pruning is mostly bearable, any other operations which result in API calls that can't be batched is super painful (though hopefully we don't hit that case often?). So overall, it's feasible, though let's see if there are other suggestions.

[dustymabe]

For Fedora Atomic Host and Fedora Silverblue we already use a split repo setup. We have a compose repo where pungi builds and stores the ostree commits and content and then we have a prod repo where content is synced when we do a release. We may be able to just re-use the existing compose repo as our "annex repo".

[jlebon]

Hmm, so I think that'd be pretty cool. Do you know how pruning works on that repo? One concern is that with all the streams outputting there, we might want some kind of semi-aggressive GC for e.g. the mechanical refs at least (though thankfully, probably the majority of files will be shared with daily Silverblue and IoT composes).

[dustymabe]

Hmm, so I think that'd be pretty cool. Do you know how pruning works on that repo?

I need to look into that.

Now that I think about it I think using the org.fedoraproject.prod.coreos.build.request.ostree-import may be kind of heavyweight for every stream that we have/want since we won't be downloading/importing just the changes, but the whole ostree every time. Maybe we should create the annex repo in s3 (using an s3fs mount may work and then just operate on it like it's local files) and store all of our streams in there (would probably need to serialize the writes). Then the org.fedoraproject.prod.coreos.build.request.ostree-import could look to see if the commit existed in the s3 annex repo first and pull from there before downloading the tarball from s3 itself.

[jlebon]

An even cleaner way would be if we were able to mount /mnt/koji in the same cluster where we run the pipeline. Then we could pretty easily do pull-local ourselves after each build.

Short of doing that, seems like there's always going to be something that will have to download the tarballs, so might as well put it near the source and avoid having to maintain a whole other repo if possible, no?

[dustymabe]

Short of doing that, seems like there's always going to be something that will have to download the tarballs, so might as well put it near the source and avoid having to maintain a whole other repo if possible, no?

yeah let's go with the org.fedoraproject.prod.coreos.build.request.ostree-import approach for now

[dustymabe]

Proposed: Re-use the existing dual repo setup that Fedora has for the Annex/Prod repos as suggested in #266 (comment)

[dustymabe]

I brought this up in the meeting today and asked for votes. I've not seen any disagreements to the proposal so far (either here or in the meeting) so I'll move forward with the proposed plan unless anyone speaks up with a good reason to deviate.

[dustymabe]

This is now done. The coreos-ostree-importer accepts import requests from the pipeline runs and performs imports into either the prod (/mnt/koji/ostree/repo) or the compose (/mnt/koji/compose/ostree/repo).

The prod repo is accessible via CDN with the following ostree remote config:

[remote "fedora"]
url=https://ostree.fedoraproject.org
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist

The compose repo is accessible via the following OSTree remote config:

[remote "compose"]
url=https://kojipkgs.fedoraproject.org/compose/ostree/repo/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/