update: distribute cincinnati-graph-data as container images

enhancements/update/distribute-secondary-metadata-as-container-image.md

@@ -0,0 +1,184 @@
+---
+title: cincinnati-graph-data-package-distribute-and-consume-in-a-container-image-repository
+authors:
+  - "@steveeJ"
+reviewers:
+  - @openshift/openshift-team-cincinnati-maintainers
+  - @openshift/openshift-team-cincinnati
+  - @openshift/team-cincinnati-operator
+approvers:
+  - TBD
+creation-date: 2020-05-07
+last-updated: 2020-05-07
+status: provision
+see-also:
+-
+replaces: []
+superseded-by: []
+---
+
+# cincinnati-graph-data: package, distribute, and consume in a container image repository
+
+## Release Signoff Checklist
+
+- [ ] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [x] Graduation criteria
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Open Questions
+
+### Content trust
+Trust in the container image registry and in the container image content is possibly an issue if we an approach where Cincinnati uses a mutable tag, e.g. `latest` tag to receive the most recent content from the container registry.
+Assuming we're using Quay as an image registry, [which does not support Docker Content Trust](https://support.coreos.com/hc/en-us/articles/115000289114-Docker-Content-Trust-support).
+If we decide that to ensure the image integrity, which means do we have which do not further complicate the workflow?
+
+### Product name for the CPaaS integration
+Which product name should this image be released under?
+Related to this is that it's currently unclear if the Cincinnati image itself will be published under its own product name or now.
+Further note: the repository for the graph-data images will not be semantically versioned.
+
+### Repository location and name
+Under which well-known container image repository will we publish the images?
+
+
+### Continuous build and release
+Can CPaaS handle continuous build and release from master?
+
+## Summary
+The [cincinnati-graph-data GitHub repository][cincinnati-graph-data] will be continuously packaged and released in form of a container image by the means of integration with CPaaS.
+This image will be available for public consumption, and will be used by the publicly available Cincinnati instances operated by Red Hat.
+The image can be transferred and served to Cincinnati instances operated by any party, including but not limited to network-restricted environments.
+The source of the container image, namely the cincinnati-graph-data, will remain publicly available and its workflows unaffected by this enhancement.
+
+## Motivation
+
+### Goals
+1. The container image will become the primary way of consuming and distributing the content of the cincinnati-graph-data repository.
+2. As long as the primary release metadata will be packaged, distributed and consumed in form of container images as well, the transport mechanism required for data consumption by Cincinnati, as well as mirroring and serving its consumed content, will be solely [Docker Registry HTTP API V2][docker-registry-http-api-v2] protocol.
+
+
+### Non-Goals
+1. Changes to the GitHub repository workflow or data format and make progress.
+2. Implementation details about the streamlining of the mirroring process for primary and secondary release metadata.
+
+## Proposal
+
+### User Stories
+
+#### Content mirroring for Cincinnati deployments
+A user might want to mirror all content which is required by Cincinnati.
+
+1. The user sets up a container image registry in the target environment.
+2. Optionally, if the target environment does not have internet access, the user downloads the release container images and the desired graph-data container image to a transfer medium.
+3. The user pushes the release and graph-data images to the target container image registry
+4. The user can now configure the Cincinnati instance in the target environment to consume the content from the available container image registry
+
+### Implementation Details/Notes/Constraints
+
+#### CPaaS Build configuration
+1. Every change to the master branch of the cincinnati-graph-data results in a new container image containing the unmodified working tree files of the new revision.
+  This will allow the plugin to continuously grab the latest version without being informed that a latest version exists.
+
+2. The image will be pushed to a publicly available container image registry under a well-known repository name, and it's `latest` tag will be updated to the latest revision.
+
+#### Plugin implementation
+1. A new Cincinnati plugin will be implemented to fetch the container image from a configured container image repository.
+  The registry, repository name, a tag or a digest will be configurable.
+  If a tag is specified, the plugin will check if the specified tag changed on each plugin execution, and download the content if that check is positive.
+
+2. The Git(Hub) scraper plugin will be deprecated in favor of the new plugin. Note that Cincinnati has support to use no cincinnati-graph-data scraper plugin at all, and can be provided the content to the filesystem directly by the its operator.
+
+
+### Risks and Mitigations
+
+### Container image content integrity
+The content of the cincinnati-graph-data repository may be altered by the process of being packaged by CPaaS and distributed via the container image registry.
+
+#### Mitigation
+A mitigation strategy is to sign the resulting image in the CPaaS pipeline, publish the signatures, and have Cincinnati verify the image after download.
+
+### cincinnati-graph-data source repository integrity
+Not strictly related to this enhancement, is the fact that we are not signing the commits in the source repository in the first place.
+By this we effectively trust GitHub not to change the content.
+
+#### Mitigation
+OpenShift architects are aware of this risk and currently do not consider it sufficiently serious to want mitigation.
+If we decide to mitigate in the future, having build tooling require commits to be signed by a hard-coded list of authorized maintainers would be one possible approach.
+
+
+## Design Details
+
+### Test Plan
+**Note:** *Section not required until targeted at a release.*
+
+Consider the following in developing a test plan for this enhancement:
+- Will there be e2e and integration tests, in addition to unit tests?
+- How will it be tested in isolation vs with other components?
+
+No need to outline all of the test cases, just the general strategy. Anything
+that would count as tricky in the implementation and anything particularly
+challenging to test should be called out.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations).
+
+### Graduation Criteria
+**Note:** *Section not required until targeted at a release.*
+
+- Maturity levels - `Developer Preview`, `Red Hat Staging`, `Red Hat Production`, `GA`
+
+##### `Developer Preview`
+- New Cincinnati plugin
+- Container image registry managed by and dedicated to prow CI
+- Integration tests using an image (to-be-determined: verifying against mocked signatures)
+- CPaaS pipeline is setup up to
+
+##### `Developer Preview` -> `Red Hat Staging`
+- Update the Cincinnati deployment template to use the new plugin.
+
+##### Red Hat Staging -> Red Hat Production
+- New plugin is successfully deployed to production
+- Adjust the Cincinnati Operator to use the new plugin
+
+##### Red Hat Production -> GA
+- Document the mirroring process for end-users
+- Definitive positive feedback from the ACM team and their use-case
+
+##### Removing a deprecated feature
+*Not applicable*
+
+### Upgrade / Downgrade Strategy
+*Not applicable*
+
+### Version Skew Strategy
+*Not applicable*
+
+## Implementation History
+
+### Initial implementation
+The initial implementation involves setting up the CPaaS pipeline, implementing a new Cincinnati plugin which downloads the published image and extracts it to disk for subsequent processing in the plugin chain.
+It also involves adjusting the Cincinnati Operator to use the new plugin by default.
+
+## Drawbacks
+
+### Increase OCP release latency
+The enhancement introduces the additional latency of the CPaaS pipeline into the OCP release workflow.
+Assuming that the pipeline is fully automated, the latency introduced is expected to be several orders of magnitude lower (minutes) compared to the OCP release cadence (hours-days).
+
+## Alternatives
+I want to clearly state that none of the alternatives mentioned below have all the same benefits as the enhancement described here.
+
+Instead of proceeding with this enhancement, we could continue to work with, or rather around, the current implementation of secondary metadata distribution in Cincinnati.
+
+Without any code changes, this would mean that we document the various ways how the cincinnati-graph-data GitHub repository can be mirrored and provided to Cincinnat, and how to reconfigure the plugin settings to directly read the data from disk.
+
+With changing the GitHub scraper plugin to be compatible with the Git protocol instead of GitHub, we could propose to run a Git server for the Cincinnati environment, and document how to configure Cincinnati to scrape secondary metadata from a custom Git server.
+
+## Infrastructure needed
+* Public container image repository
+* CPaaS build system with push access to the aforementioned container image repository
+
+[cincinnati-graph-data]: https://github.com/openshift/cincinnati-graph-data
+[docker-registry-http-api-v2]: https://docs.docker.com/registry/spec/api/