Rebuild Build-Release process: Bintray is going away (release pipeline)

[planetf1]

Just received an email from JFrog stating

Thank you for supporting JFrog through your use of Bintray. We are and continue to be a community-focused company, and in the spirit of transparency we’re giving you advance notice that as of May 2021, we will be sunsetting Bintray.

See https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/?utm_source=mkto&utm_medium=email&utm_campaign=bintray-sunset&utm_content=global-02-2021

In fact the key point is:

When | What’s Changing
-- | --
Now | Bintray, JCenter, GoCenter, and ChartCenter service continues.Bintray/JCenter users should start migrating to a new hosting solution
February 28th | No more submissions will be accepted to Bintray, JCenter, GoCenter, and ChartCenter

Currently our Azure release pipeline uses bintray as a staging point for sync to maven central. These artifacts start on artifactory (JFrog) in any case, so I would hope the change is minor, but this needs investigation prior to release 2.7 .

We do not require or depend on bintray on any permanent basis beyond the pipeline.

This publishing process has been fraught with issues on an ongoing basis -- mostly around validation & performance/reliability so it would be prudent to look at alternative routes

Also worth pointing out that Azure pipelines (which we use for release) vs GitHub actions (which we use for PR, and some of the new repos) is not really a factor here other than in ease of integrations

cc: @mandy-chessell

[planetf1]

Raised support request https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21485 with the Linux Foundation

[planetf1]

Correction - Bintray AND JCenter are going away.

This is more significant --
a) Do we continue pushing to maven central
b) If so what process do we use

Also if artifactory is no longer required (or easier -- as it was up until now) then looking at github artifact support may also be sensible

[planetf1]

Github has quite a bit of documentation. It's integration recommendations are more based on standard, built in, maven behaviours rather than a complex pipeline.

https://docs.github.com/en/actions/guides/publishing-java-packages-with-maven#publishing-packages-to-the-maven-central-repository-and-github-packages shows an example pom where artifacts are published by a build to both github's own respository as well as maven central (though our large number of artifacts may cause problems)

The example also shows how this release process can be linked into the 'release' process - ie creating the release on github causes this publish to then occur

There would also be the potential for triggering automated pre-release tests

It might also be possible to use other actions to do the pre-release steps such as version updates - manual tasks that take time each release

The github package repo could be used for all builds to serve -SNAPSHOT versions for development purposes.

This kind of integration seems less errorprone and less manual work

[planetf1]

In terms of mitigation... If a pipeline is not ready our options are

• Do not do a Feb release - I think this is undesirable
• Run the release process as is which should publish to Aritfactory. Direct users via blog/tweet/release to use this repo
• In addition to b) or instead of, manually script up a mvn publish (getting the signing etc done may be the hardest) - we did do this in the past..

[planetf1]

Posted to solicit input at https://lfaifoundation.slack.com/archives/C01KXK9Q142 (go to https://slack.lfai.foundation/ if you'd like to join - open to all).

[planetf1]

Note - there is some background to other issues with the current process in #3194 - now closed

[planetf1]

I've had feedback that the LF team has transferred the issue to https://jira.linuxfoundation.org/browse/RELENG-3387 but no specifics yet

A few thoughts after a little investigation

• As a dev team we find github actions easier to work with than Azure pipelines
• Github now offers support for package repositories (for maven) of arbitary names
• We have no real need for bintray/JCenter in their own right - it was only a pathway. Though we did add JCenter as an alternative source for 3rd party packages to improve build reliability (if maven central down)
• Artfactory is rich in function, but we don't really use it.

My current proposal would be

• Move to using github actions for the release pipeline
• Do not store artifacts for PRs (additional time processing) [we could later if needed]
• Store snapshots for every merge build to a github package repo (no artifactory etc) ie egeria-snapshot
• Publishes release to a github package repo ie egeria-release
• Additionally publish release to maven central in a transactional way, using a staging directory for the entire release - not per artifact as is done currently by JCenter, and likely a large part of the performance/reliability issue

This would remove dependence on all JFrog tooling, and even provide mitigation against maven central issues, since the packages would still be available at github (and if github fails we have a lot more to worry about)

In more detail our release process currently includes

• branching 'master' to a release egeria-release-xx
• stripping release notes & replacing with link
• updating version in poms to remove -SNAPSHOT
• incrementing version in master for (release)+1-SNAPSHOT
• testing! (additional manual tests - in future longer automated tests)
• publishing to release repo
• creating github release (including copying release notes)

Additional automation here would be useful, but a good starting point would be to make the publishing to an upstream repo reliably. Currently this can take 11+ retries over many days. Fixing that single step with simpler technology would be a huge bonus. Further my using a developer-friendly ecosystem (like github actions) the other features could be incrementally added. There are already multiple examples of each of these steps supported by examples and additional marketplace actions.

For the publishing to maven central we could take two approaches

• Publish both to github actions & ossrh as part of the same action - this is the bases of the standard github docs
  OR
• Develop a 'sync' approach

I'm more inclined to the latter as it decouples our native github work with a separate action having dependence on a third party that we know has been unreliable (outages, performance) but if we find ossrh is reliable it may not be needed

Also worth noting

• Most comment here refers to the egeria core github repository. We now have additional repositories and are in the process of refactoring. These other repos contains non-maven artfacts in many cases. Orchestrating some cross-repo changes via actions could be useful (ie release numbers, starting the release process...). Again here I think having a flexible ecosystem is the required starting point & we can evolve in future.

[planetf1]

On repositories - Using github packages first, and then any upstream as secondary (for release publication) seems a good approach that would cover maven/gradle, docker, npm- making it possibly to have a single source for all egeria artifacts (github) both packaged, source tar.gz, source repo, issues etc. Then additionally we can push to the 'native' repos for the artifact type.

[planetf1]

Have updated LF team with a summary and linked back here.

[cmgrote]

Adding myself to track for applying the same changes to connector repos

[cmgrote]

This appears to work for publishing snapshots to GitHub packages anytime code is merged into main / master branch: https://github.com/odpi/egeria-connector-crux/blob/main/.github/workflows/maven-snapshot.yml

Two important "gotchas":

• Need to ensure that we define a github ID'd entry in the distributionManagement section of the pom.xml, as this is what the action uses to determine the target Maven repository to publish into
• Need to ensure that settings.xml is added to the .gitignore of the repository, as the action will generate a settings.xml with the appropriate token details to allow the package publication -- and this generated settings.xml will not have an SPDX header (and will therefore fail the Maven build for the package against the RAT license checks)

[cmgrote]

Also note some downsides to GitHub's packages (vs. Artifactory):

• There does not appear to be any general URL that automatically resolves to the latest snapshot, if a user simply wants to directly download a Jar file (without the overhead of Maven itself). An example URL: https://maven.pkg.github.com/odpi/egeria-connector-crux/org/odpi/egeria/egeria-connector-crux/1.0-SNAPSHOT/egeria-connector-crux-1.0-20210210.160820-1-jar-with-dependencies.jar (note that the full timestamp details must be included, simply short-handing to -SNAPSHOT does not work)
• Downloading a package, even via Maven, and even for public repositories, requires authenticating against GitHub. While this should hopefully be a one-time setup step, it does add complexity to e.g. workflows as it means needing to capture tokens or other secrets in every repository that needs to make use of the snapshot version, so that these sensitive details can be passed through environment variables to generated settings.xml files, to be used as part of the overall build / deployment pipeline.

The two of these combined basically make it impossible to just point users at a jar file they can download to quickly get up and running: users will need to install Maven and be familiar enough with its usage to setup authentication against a non-standard repository (GitHub)...

[planetf1]

Thanks @cmgrote. The first response are I think useful tips. For the last two the latter is the main problem with github packages as a repo

The essential change we have to make is away from bintray/JCenter as a vehicle to get our artifacts published for a release to maven central (assuming that remains the ultimate destination).

That in itself should be very viable as we can push to sonatype direct from a pipeline.

If we are doing any work in this area, moving to github actions seems logical as it is IMO better suited for the broader dev team to work with, and there's a large & evolving ecosystem. So more automation in the future

Removal of artifactory isn't essential -- there is support in actions to still publish to artifactory should we wish, and artifactory isn't going away, but it was an idea that perhaps we could simplify alltogether by skipping.

It's also worth noting this specific issue is about the release pipeline -- so in this context artifactory really is optional in any case as it's just used for staging... your (very good) point is more about the merge pipeline -- which I do think logically should move to actions - there's an issue open on it.

Maybe we can't remove it's use there. . I think the interesting part will be what we see from performance of pushing 400 artifacts (there's other issues open on trying to consolidate that), if we do it within a single transaction. If it's not too bad we could push snapshots there - whether on PR, daily etc. If it's bad, we need to use something else so could go with artifactory
I think the first 'big idea' here is to move from azure pipeli

[planetf1]

I should add that these are high level thoughts. I've bounced a few ideas with the LF team - though they are new to the area, & are also sharing with some other LF AI & Data projects. I've also contacted sonatype (maven central) to get guidance on testing/staging. I'll likely start on the actions work around 22 Feb (vacation prior)

[cmgrote]

Agree fully on the move towards GitHub Actions -- and fine with living with some of the potential short-term pain it could introduce (on snapshot packages for the merges). Just to further add to that "pain", though, I've just further noted that publishing a new snapshot appears to take some time to even appear on GitHub's packages page even though it's accessible by a direct URL if you know the precise timestamped set of jar files that were actually created (for example, by reviewing the GitHub Action build log).

So even asking users to go to the packages page of GitHub and manually find the correct jar isn't really feasible at the moment -- and I suspect would never be feasible for Egeria core itself (as the page lists all artifacts: all jars, poms, etc -- and all duplicated per timestamped-snapshot). Even for the connector repos, which generally only have 1-2 jars, this list becomes 10s of entries long after just a couple of merges into the main branch -- this, for example, is just 2 snapshots (3rd has not shown up yet) for a 1-artifact-repo: https://github.com/odpi/egeria-connector-crux/packages/617324)

[cmgrote]

Just for completeness, there are some hacks suggested (by GitHub itself) to allow public access to the packages, on this thread: https://github.saobby.my.eu.orgmunity/t/how-to-allow-unauthorised-read-access-to-github-packages-maven-repository/115517

(Basically you're just creating an access token and then giving that token to the public.) Other threads suggest that proper anonymous package access is coming sometime, though the original issue seems to have been raised in 2019 and the only timeline response I saw was "probably not any time in 2020". Now that we're in 2021, maybe it'll be there sometime soon 🤷

[planetf1]

Initial pipelines are being added (still testing)

• Release - This is triggered by either a manual action (for testing) or creation of a GitHub release. This replaces the release pipeline on azure pipelines.
• Merge - this will replace the merge/docker/windows builds on azure pipelines

Other pipelines have been cleaned up

Snapshots are published from a master merge build to maven central snapshot repo
Releases are published to a staging area - which will need to go through close() release() on oss.sonatype.org - intentially to allow for verification

Benefits over prior implementation:

• Release pipeline kicked off automatically when creating the release on github
• Atomic release - staging area is verified & released in one go
• Improved performance. hopefully radically from ~3 days to hours due to use of single staging area
• Github actions more familar to broader team
• easily extensible for future needs (release notes, javadoc publishing etc)

Still to do

• Moving the docker build across (and will use github actions not encapsulated in maven) and add into merge build
• Adding a windows 'merged' build on a schedule
• using proper lfai credentials & review/approval of security of these credentials
• Fixing up 'bad' poms which would block release
• updating release documentation
• team review

Caveats

• Jobs will be disabled (but not deleted) in azure pipelines as the corresponding version runs here (to avoid confusion)
• no publishing currently to either artifactory or github
• need to work with other repo owners to ensure things tie up
• still testing ....

I am unable to fully test the release without creating a branch to work with. Given 2.7 is imminent I will leave that until then .

For any 'bad' poms they will need to be corrected, or the following should be added to their pom:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-deploy-plugin</artifactId>
  <version>X.Y</version>
  <configuration>
    <skip>true</skip>
  </configuration>
</plugin>

[planetf1]

The snapshot publishing seemed to work well. The last merge build took 37 minutes for a full maven build, and a publish of all artefacts to maven central (snapshot repo)

No validation on Pom correctness was done as expected

This bodes well for the release albeit with the extra checks adding much more time , but should be orders of magnitude better than prior release pipeline performance

[planetf1]

I ran through some of these changes in a sandpit project, but it's somewhat tricky to really verify thoroughly without making them on the main codebase. This has been done, so now I'm after feedback to see what is broken - if anything ..

@cmgrote I believe your connectors build against the last released version of egeria - so as consumer you are unaffected. However we need to figure out what makes sense for distributing the connectors themselves. I think we could follow the same pattern in your repo repositories . I suggest removing any references to jcenter, artifactory . Finally I know the vdc chart was using artifactory. I've opened odpi/egeria-samples#32 but note there are other breakages in that area, and planned moves of various code between repos

@wbittles the postgres connector repo is currently consuming egeria snapshots. Your gradle build definition may refer to jcenter and/or artifactory. If so I suggest you remove and just use maven central, from where you should get up to date snapshots. The publishing of maven artifacts will likely (?) need to follow the pattern done here once the connector is updated and we want proper ci/ci in place . odpi/egeria-database-connectors#16

@sarbull I note the egeria-ui is currently using azure pipelines (which I initially created). It's worth considering moving this over to github pipelines odpi/egeria-ui#47

@lpalashevski I believe you consume maven artifacts for deployment. If you are using the odpi artifactory you'll need to use maven central instead from now on.

Finally, I would hold off finalizing any changes until the LFAI team have updated credentials and/or refactored anything they need do. Currently the credentials are scoped at egeria only as I don't have repo level permissions, though could setup individually as needed. Would also appreciate any comments on what has been done so far & confirm whether it makes sense

[planetf1]

The windows daily build is now running correctly -> https://github.com/odpi/egeria/actions/workflows/windows-daily.yml
The merge build is correctly updating snapshots on MC, and docker images on docker hub -> https://github.com/odpi/egeria/actions/workflows/merge.yml
The release pipeline is not fully tested, but will be used for 2.7 and fixed as needed -> https://github.com/odpi/egeria/actions/workflows/release.yml

[planetf1]

I contacted sonatype for clarification on how snapshots are managed cc: @davidradl

There is a scheduled task on oss.sonatype.org that runs once a week. It keeps at least 3 snapshots. If you have more than 3 snapshots, it will retain up to 2 days' worth of snapshots. The job will also delete snapshots that correspond to a release version, however, there is a 10-day grace period before those deletes happen.

That sounds fine for us.

[planetf1]

See also #4734 for the 2,7 release

The pipeline appears t run perfectly :-)
Now closing the repo to get the pom validation underway

[planetf1]

Pipeline failures are actually on signature currently. see https://gist.github.com/475bbbc686ea6a928a2fc18ea83258ba

This seems to affect every artfifact, so presumably I omitted a step relating to signing

[planetf1]

Most signatures are in fact now correct after the prior fix, however a few fail:

typeId | signature-staging
-- | --
failureMessage | Missing Signature: '/org/odpi/egeria/open-metadata-docker-configure/2.7/open-metadata-docker-configure-2.7-docker-info.jar.asc' does not exist for 'open-metadata-docker-configure-2.7-docker-info.jar'.
failureMessage | Missing Signature: '/org/odpi/egeria/open-metadata-docker-jupyter/2.7/open-metadata-docker-jupyter-2.7-docker-info.jar.asc' does not exist for 'open-metadata-docker-jupyter-2.7-docker-info.jar'.
failureMessage | Missing Signature: '/org/odpi/egeria/open-metadata-docker-egeria/2.7/open-metadata-docker-egeria-2.7-docker-info.jar.asc' does not exist for 'open-metadata-docker-egeria-2.7-docker-info.jar'.
failureMessage | Missing Signature: '/org/odpi/egeria/open-metadata-docker-uistatic/2.7/open-metadata-docker-uistatic-2.7-docker-info.jar.asc' does not exist for 'open-metadata-docker-uistatic-2.7-docker-info.jar'.

These are all docker projects so the pom is clearly slightly bad

[planetf1]

These projects are defined with packaging=pom - but we have many others that do this (where they don't supply any artifacts). It's possible the signature handling is being overriden by some of the plugins (spotify docker etc) that we use for this build - or that it is something to do with extra '-info' files (since we do see .asc files for the pom). Also we don't really have a 'jar' built here in any case. This demonstrates the slightly awkward fit where we are using maven to build non. maven like artifacts, more like a makefile/gradle build than a maven artifact

We intend to move the docker builds out of maven artifacts - or at least some - for simplicity. Rather than doing this refactoring now, and instead of figuring out how the plugins are impacting the signatures, a pragmatic approach is to skip uploading of these artifacts - since the actual files produced by these artifacts are actually sent to docker hub, and the only 'source' is a dockerfile.

[planetf1]

The release pipeline has now been able to upload egeria artifacts correctly.
Furthermore when 'close' is hit on oss.sonatype.org the validation completes successfully meaning the artifacts are ready to release.

This pipeline was started manually from the github action UI, but should also trigger from creating a github release
It is possible to automatically release without logging in to oss.sonatype.org - but for now leaving that as manual just gives an opportunity to check the release beforehand. It only requires 2 clicks and a few minutes to to this last step. In future we could do it automatically (perhaps alongside automated tests of the staging repo)

[planetf1]

Opened #4683 to update docs on using the release pipeline
Implementation is otherwise successful and first used on #4734
An IT issue is open to see if LF support can use their credentials, but for now we go with what we have