feature: Support the new Sonatype Central API

[Andrapyre]

Description

This PR adds support for the new Sonatype Central publishing api, documented here and here. To use, ensure that your environment variables contain the appropriate credentials and add the following to your library build.sbt:

import xerial.sbt.Sonatype.sonatypeCentralHost

ThisBuild / sonatypeCredentialHost := sonatypeCentralHost

New Dependencies

Adding these was necessary, since the airframe client does not support multipart uploads. Perhaps it is possible to custom-wire the airframe client so that it does this, but I was not successful in an initial try. There is, in any case, no airframe documentation about multipart uploads, nor could I find any support in the code base. The following dependencies made implementation very easy. I have added the "effect-free" versions, since the current sbt-sonatype codebase does not use any effects, though use of effects would make certain things like resource management easier.

"com.softwaremill.sttp.client4" %% "core" % "<version>",
"com.softwaremill.sttp.client4" %% "circe" % v"<version>",
"com.softwaremill.sttp.client4" %% "slf4j-backend" % "<version>",

New SBT Keys

Users can customize Sonatype Central deployment names like so:

sonatypeCentralDeploymentName := DeploymentName.fromRawString("<custom-deployment-name>")

Additional Notes

1. This initial version hard-codes the bundle upload to a "user-managed" deployment - i.e. after the bundle is uploaded and verified, users will need to manually press "publish" in order for it to be published to Maven Central. This functionality is ideal for a release candidate version of sbt-sonatype, as it gives users the ability to verify that everything is working before publishing. A final version can either hard-code the deployment to "automatic" or introduce a new sbt key so that users can specify this themselves (preferred).

2. Sonatype Central currently does not support publishing snapshots. The documentation says that an uploaded (but not published) deployment can be tested in production by referring to the deployment id, but I have not verified this.

3. I deployed a library to Sonatype Central using these new changes and it worked with no issues.

[Andrapyre]

@xerial , before finalizing and adding tests, could I get an overall thumbs-up on this implementation? Would like to make sure that there's nothing controversial here, otherwise time spent on testing could be wasted effort.

[Andrapyre]

Formatting issues are fixed - all workflows should now succeed. If there are no issues with the overall approach, I'll add some tests and finalize.

[xerial]

@Andrapyre First of all, thanks for preparing this PR. It gives us a head start. I also could learn a lot about the new Sonatype API.

Major and unknowns:

• Can we merge two similar keys, sonatypeCentralDeploymentName (how this will be used in the new API?) and sonatypeSessionName? sonatypeSessionName is used for creating a unique session in the legacy API, so that we can resume the release process even after network failures.
  • Or make it just a sonatypeDeploymentName (String). I believe there is no need to expose the internal DeploymentName model class to users. As central will eventually be the default repository, we can remove Central from the key name.
• It depends on the plan of Sonatype for migrating legacy API users to the new one, but if the process can be totally different, instead of merging sonatypeBundleRelease, it would be better to create new separate commands like sonatypeRelease (for automatic deployment) and sonatypeUpload (for user-managerd deployment), so that sonatypeBundleRelease can call sonatypeRelease internally and the user can choose what they need.

Good

• Creating SonatypeCentralClient extracting some common code look good.

Minor, but can be painful for me in the long run.

Note: You don't need to tackle the items mentioned here, as I will be able to manage them.

• Coding style differences, especially using Either for error handling. I understand the benefit but I've also observed it's been confusing people who are relatively new to Scala's ecosystem.
  　- Please note that I've been maintaining this plugin for almost 10 years. While using sttp is acceptable, introducing circe might not be the best choice, especially considering that the main creator has left the Scala world. If you require a JSON parser, I recommend using airframe-codec, which is already used within airframe-http for parsing JSON responses.
• If airframe-http supports multi-part upload, we can use the same retry mechanism without introducing sttp, circle, Either, which is beneficial in the long run. I'll think about supporting multi-part upload in airframe-http. http-client: Support multi-part upload wvlet/airframe#3420

After receiving your responses, I can suggest further adjustments to the PR, or I can make these changes myself to save your time, depending on your preference.

[Andrapyre]

@xerial , thanks for the quick response. Several points:

1. The deployment name is purely cosmetic in Sonatype Central, so that the user can meaningfully distinguish in the ui between different deployments. Multiple deployments can even have the same name and if no name is provided then they will automatically be assigned the filename (in this case bundle.zip) as the deployment name. To avoid the scenario where the user logs in and sees 100 "bundle.zip" deployments with no way to figure what is what, I've set the deployment name to "<org_name>.<artifact_id>-<version_name>", but left it open to users to customize, hence why the class is exposed. Each deployment comes with a deployment id and that would probably be the equivalent of the session name (though I don't know, since I haven't worked with the old api). In any case, this implementation gives the deployment id its own class and it is not exposed to the user to customize, since it's generated by Sonatype Central.

2. Downstream plugins (such as typelevel sonatype and sbt-ci-release) use the sonatypeBundleRelease command. To give users the freedom to specify whether they want their deployment to be managed or automatic without needing to make downsteam changes, I would introduce a new key, isSonatypeReleaseAutomatic and have sonatypeBundleRelease pass the value down accordingly, while keeping the rest of the functionality as is in this PR.

3. I will make a best effort attempt to migrate to airframe codec. It should be pretty easy. If it gets time-consuming or I run into larger issues, I will leave as-is or pick a different framework, if you prefer (jsoniter-scala is very fast/the fastest and well maintained - I've had a very good experience with it in other projects). I agree in general that fewer dependencies are better and would have preferred to introduce none with this feature. I chose circe in part because it has first-class support in sttp.

4. I appreciate that Scala can be difficult to read and work with, especially for new beginners. In my experience though, that generally applies to use of effects - I haven't seen anyone complain about simple types like Either or Option. Are you referring to my utils package with the leftMap implementation? I can add a few comments so that it's clear what those are practically doing. Otherwise, I find using Either to wrap exceptions far less messy than putting try/catch everywhere and would prefer not to change it if possible.

Let me know your thoughts. I'll proceed in any case with revising the json parsing away from circe.

[Andrapyre]

One more thought - I wouldn't hold off on merging this until airframe provides multipart support. There are users that need this functionality ASAP (including me) and I'm not sure that multipart uploads, including support for octet streams, will be a quick and simple feature add. Could we merge first and then come back and refactor when airframe provides multipart support?

[xerial]

@Andrapyre The internal implementation can be changed later, so merging this first works. You need to address only two points addressed as major here #474 (comment)

1. sonatypeDeploymentName : String. // Do not use DeploymentName object as the setting type. Plain String should work here
2. sonatypeBundleRelease
   • Call sonatypeRelease sonatypeReleaseCentral (automated release at Central) internally
   • Add a separate command sonatypeUpload (for manual release)

I'm against changing the behavior of sonatypeBundleRelease with an option (e.g., isSonatypeReleaseAutomatic ) because sonatypeBundleRelease is supposed to complete all of the deployment steps at once. These command names and behavior will be difficult to change once the users start depending on the behavior.

See also existing commands for supporting such step-by-step executions https://github.com/xerial/sbt-sonatype?tab=readme-ov-file#individual-step-commands I believe not so many users are using these commands, though.

[Andrapyre]

Understood - I'll make said changes and revert.

[Andrapyre]

Alright. Several changes here:

1. The sonatypeDeploymentName setting is now a String type.
2. I migrated to zio-json which has a similar structure to circe and has better support/maintenance long-term. I didn't see airframe codec or jsoniter-scala easily decoding the deployment stage enum we have, something that is very simple with both circe and zio-json.
3. Two new commands - sonatypeCentralRelease (automatic) and sonatypeCentralUpload (user-managed) sonatypeBundleRelease calls the same function as sonatypeCentralRelease if the credential host is Sonatype Central.

I will add some tests (mainly for the Sonatype Central client), along with documentation, and then request a final review.

[xerial]

Apologies for the delay. I'll be looking into this next week. And with Sonatype's recent announcement about account migration, it looks like I'll also need to get my repositories set up there.

[Andrapyre]

All good - I had a delay myself due to other commitments taking priority. I did manage to test automatic deployments today, as well as a cross-versioned deployment to Sonatype Central, and everything worked as expected. I will forego tests for now for the sake of immediately merging this. The other client doesn't have testing as is and it seems that sttp might be ripped out in the near future and then all tests would need to be rewritten anyway. On that subject, if the plan is to keep sttp, I'm happy to add tests in a follow-up PR.

I still need to add some documentation and push all changes.

[Andrapyre]

Documentation is added. Happy to receive a review now.

[Andrapyre]

@xerial , I have moved all of the basic client operations into a client library here, where I will maintain and test it long term. I also updated some of the retry logic to immediately fail when it encounters user errors (e.g. the user specified wrong credentials, so we shouldn't retry for the next 30 minutes, etc.).

Could we get a 4.0.0-RC1 out in the next few days, so that the community can test this? Main question mark for me is how very large, cross-built projects with a lot of modules will be affected and whether we need to build out any extra support for such projects. With Sonatype Central, users can only upload one large bundle. It looks like the legacy oss environment supported parallel processing of some kind vis-a-vis this comment here? The same functionality isn't available in Central. Not sure about the potential breakage there.

[KacperFKorban]

@xerial Are there any blockers before this can be merged and an RC can be published? It would be amazing to be able to use sbt-sonatype with Sonatype Central.

BTW, thanks for maintaining this library ❤️

[Andrapyre]

@KacperFKorban , @xerial expressed concerns offline that there was too much overlap between legacy and newer functionality, when many of the features are quite different or mean totally different things. For instance, there are no staging repos, host names, snapshot repos, or profiles, etc. in the new publishing api. Nevertheless, working the new central portal api into this plugin requires us to set values for all or most of these, which makes the code harder to reason about and could lead to potential bugs down the road.

When I implemented this for mill the team requested that this functionality be kept in a completely separate lib, though I also initially integrated it into their existing sonatype publishing lib. I've decided to take a similar route here and put this into a separate plugin, which I will maintain here. I'm having some issues deploying it - see lumidion/sbt-sonatype-central#1. When these are resolved, I will deploy the plugin, close this pull request, and request that the corresponding issue be closed as well.

[xerial]

Hi folks. As I don't have much time to review or fix it, I'll ship it now and fix it (or review it) later.

[xerial]

Released 3.11.0 https://github.com/xerial/sbt-sonatype/releases/tag/v3.11.0