Could a GitHub release replace a build manifest, should we try it?

[peternied]

Build manifests as are used in OpenSearch / OpenSearch Dashboards contain a lot of information, is all that information necessary or is a GitHub release and associated artifacts enough?

Acceptance Criteria:

• Determine if common use cases can fallback to a release driven datastore rather than a manifest file.
• List Pros and cons of the to using manifest file or github or other possible solutions
• Review with stakeholders to check if the solution is feasible
• Update the design to allow github releases (pre-release) as the triggering data source

[peternied]

After doing a little digging, the release supports a direct publish mode, and a draft that can be published later. The draft release can take the intended tag name and when the release it published it would automatically be created. By using a template, this can be pre-populated.

Releases can be editing afterwards, but the tag pointer possible, it would have to be switched. While releases could be driven by this tool, the pre-release repository is not accessible publicly so additional access would be needed for each repo that supports this.

To use the release feature as part of github, I would advise a system that:

• Repo is setup to release on demand
• Maintainer wants to make a release
• They create a draft release, template includes fields that might / might not be used for releasing
• Periodic scan for unpublished releases is found
• Run the release workflow in the repo
• If the repo successes, publish the release
  • This automatically creates a release tag at the commit that was pushed
  • Allows opportunity to associate artifacts to the github release page
  • Allows opportunity to include links/refs/notes that are release specific

If there are details that need to be managed in a format in-line with the existing input manifest model, a shared github action could create manifest file after a release draft is created.

Potential areas of concern

• How will maintainers know not to hit 'publish' and instead rely on the deployment workflow?
• If there is a release failure, how does the team find out?
• Each repository will need to allow the workflow system permissions to scan for draft releases.

Github action on release
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release

Release object reference for github
https://docs.github.com/en/graphql/reference/objects#release

Creating templates for release notes:
https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes

Api reference release notes:
https://docs.github.com/en/rest/reference/releases

[gaiksaya]

The GitHub release seems like a better option that makes the client release a push based model. Removing the manifest file completely. The GitHub release needs to be created/drafted for the jenkins release workflow to run. The trigger payload gives the necessary details that can be used by the jenkins lib functions https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release
As stated above release assets can be used for releasing.

For example, the release workflow is triggered on tags, the high level end to end workflow would look like:

In the long run, this is low/no maintenance compared to manifest files. The jenkins File used for releasing can differ the source of artifacts used for publishing (download from s3 / GH release / any other source)

Note: Draft release can also be replaced by pre-release on GitHub that is publicly visible.

[peterzhuamazon]

The GitHub release seems like a better option that makes the client release a push based model. Removing the manifest file completely. The GitHub release needs to be created/drafted for the jenkins release workflow to run. The trigger payload gives the necessary details that can be used by the jenkins lib functions https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release As stated above release assets can be used for releasing.

For example, the release workflow is triggered on tags, the high level end to end workflow would look like:

In the long run, this is low/no maintenance compared to manifest files. The jenkins File used for releasing can differ the source of artifacts used for publishing (download from s3 / GH release / any other source)

Can you explain a bit on what draft a release means here?

[gaiksaya]

Can you explain a bit on what draft a release means here?

@peterzhuamazon See https://github.com/marketplace/actions/gh-release
You can decide if you need to create/draft/publish/delete a release on Github https://github.com/marketplace/actions/gh-release#inputs

This is what is draft release looks like:

It won't be publicly visible until you publish a release.

[bbarani]

@gaiksaya I like this solution but does this tie our release process with GitHub repo and tools? I am good with this model but want to make sure that the process can be easily reproduced outside of GitHub as well.

[gaiksaya]

It does not directly tie it to GH but since our repos, their CIs are on GH it makes sense if part of it is tied. The jenkins libraries that will take care of the actual release will be independent of GH rather GH is a bridge that will trigger, (maybe bring the artifact to) jenkins.

[dblock]

+1 on the staging area being GitHub. I think this will work at least for Java (Maven) and Ruby (pkg) artifacts.

[prudhvigodithi]

Hey, few thoughts:

• One option is, the build and compile should completely happen on Jenkins, so the artifacts can be shipped from Jenkins workspace to the external repos.

• The build and compile code can run on gitHub workflows but again should be shipped to a central place (s3, minio or any other option), where Jenkins can pick the artifact and just run the code to publish the artifacts, with this I see management of jenkins, agents and shared libraries will be minimal, as the heavy lifting is being done with gitHub workflows, jenkins is just used to ship the artifact.

• Github has an option to store artifacts on the repo itself https://git-lfs.github.com/ (should explore more on this) where jenkins can pull the built artifacts directly.

[dblock]

Please note that git LFS is not free, and you can easily DDoS someone else's account.

[gaiksaya]

We will be going ahead with GitHub release. For artifact staging, if release assets are not sufficient we can look into bringing up the infrastructure to support additional storage. Updated the design doc with the above approach.
Thanks!