KPM fails to install SNAPSHOT version of plugins from Github packages

[nick-at-finix]

Hi team, we are using kpm to install our plugins from a private Github repo's packages. We have no issue installing release versions, like so:

kpm install_java_plugin com.company:hello-world-plugin \
    --overrides url:https://maven.pkg.github.com/company/hello-world-plugin \
      token:ghp_... \
    --group-id com.company \
    --artifact-id hello-world-plugin \
    --version '2.0.1'

result:

Artifact has been retrieved and can be found at path: /var/tmp/bundles/plugins/java/hello-world-plugin/2.0.1/hello-world-plugin-2.0.1.jar

However, when we try with a SNAPSHOT version:

kpm install_java_plugin com.company:hello-world-plugin \
    --overrides url:https://maven.pkg.github.com/company/hello-world-plugin \
      token:ghp_... \
    --group-id com.company \
    --artifact-id hello-world-plugin \
    --version '2.0.1-SNAPSHOT'

This is the result:

W, [2024-05-21T14:03:00.570384 #4048]  WARN -- : Unable to retrieve coordinates {:group_id=>"com.company", :artifact_id=>"hello-world-plugin", :packaging=>"jar", :classifier=>nil, :version=>"2.0.1-SNAPSHOT"}: The artifact you requested information for could not be found. Please ensure it exists inside the Nexus.
The artifact you requested information for could not be found. Please ensure it exists inside the Nexus.
/opt/kpm-0.11.1-linux-aarch64/lib/vendor/jruby/2.6.0/gems/kpm-0.11.1/lib/kpm/nexus_helper/nexus_api_calls_v2.rb:162:in `process_response_with_retries'
/opt/kpm-0.11.1-linux-aarch64/lib/vendor/jruby/2.6.0/gems/kpm-0.11.1/lib/kpm/nexus_helper/nexus_api_calls_v2.rb:143:in `get_response_with_retries'
...
/opt/kpm-0.11.1-linux-aarch64/lib/vendor/jruby/2.6.0/gems/kpm-0.11.1/bin/kpm:8:in `<main>'
org/jruby/RubyKernel.java:1052:in `load'
/opt/kpm-latest/lib/vendor/jruby/2.6.0/bin/kpm:23:in `<main>'

I can confirm that this package does indeed exist, and can fetch it via curl, so it is not a connectivity or missing package issue. Please let me know if I can provide any additional details.

Related code:

• https://github.com/killbill/killbill-cloud/blob/8ba1ad87fb2f8d0c46f56a6053c3c8447117fd03/kpm/lib/kpm/nexus_helper/github_api_calls.rb

[pierre]

I remember GitHub support for Maven being quite finicky and non standard...

Are you able to pin point part of the code that needs updates? Try with KPM_DEBUG=1.

[nick-at-finix]

Hi pierre,

Thanks for the quick reply, and the debugging tips. From a quick glance at the code, I thought that kpm somehow used maven to download the artifact, which would resolve timestamped snapshots. Alas, this is not the case.

The url for my snapshot artifacts is suffixed with a timestamp, which cannot be removed in maven 3 (see here).

An example of the URL for my sha1 file: https://maven.pkg.github.com:443/company/hello-world-plugin/com/company/hello-world-plugin/2.0.1-SNAPSHOT/hello-world-plugin-2.0.1-20240520.203819-1.jar.sha1. The existing ruby code would not be able to "reconstruct" this url from the given maven coordinates only.

[pierre]

Right, we need the server to tell us the filename.

This is what we do for other Maven repositories (e.g., Cloudsmith):

killbill-cloud/kpm/lib/kpm/nexus_helper/cloudsmith_api_calls.rb

Line 18 in 8ba1ad8

# For SNAPSHOTs, we need to figure out the version used as part of the filename

Does GitHub provide the SNAPSHOT filename in maven-metadata.xml? If so, it should just be a matter of porting that code into the GitHub adapter.

If you have an environment on GitHub with SNAPSHOT packages, it's quite easy to write an integration test for it to test things out, see #188 for an example.

[nick-at-finix]

Here is what the maven-metadata.xml looks like:

<metadata>
	<groupId>com.company</groupId>
	<artifactId>hello-world-plugin</artifactId>
	<versioning>
		<latest>2.0.1</latest>
		<versions>
			<version>2.0.1-SNAPSHOT</version>
			<version>2.0.1</version>
		</versions>
		<lastUpdated>20240422151152</lastUpdated>
	</versioning>
</metadata>

[pierre]

Is this for https://maven.pkg.github.com:443/company/hello-world-plugin/com/company/hello-world-plugin/2.0.1-SNAPSHOT/hello-world-plugin-2.0.1-20240520.203819-1.jar.sha1?
Or I'm guessing this would be for https://maven.pkg.github.com:443/company/hello-world-plugin/com/company/hello-world-plugin/2.0.1-SNAPSHOT/hello-world-plugin-2.0.1-20240422.151152-1.jar.sha1

If so, it looks like we'll need to manipulate a bit the metadata to construct the filename...

Is your company able to provide a patch? If not, we'll add it to our backlog (not sure when we'll get to it though).

[nick-at-finix]

Or I'm guessing this would be for https://maven.pkg.github.com:443/company/hello-world-plugin/com/company/hello-world-plugin/2.0.1-SNAPSHOT/hello-world-plugin-2.0.1-20240422.151152-1.jar.sha1

Yes, sorry, bad copy/paste/obfuscation on my part.

Is your company able to provide a patch?

I would love to, however I do not know ruby at all, and do not have time to invest in this, sadly.

If not, we'll add it to our backlog (not sure when we'll get to it though).

Totally understand, I also maintain OSS in my free time, and know that resources are at a premium.

If/when I am able to contribute, do I need to do anything aside from sending a PR with code update + test?

---

Fwiw, it is looking like we may need to investigate the nuances of Github's packages a bit more; I am thinking that <lastUpdated>20240422151152</lastUpdated> might also reference the released version if that is published after the latest snapshot :(

Edit: I can confirm that lastUpdated is indeed the timestamp of whatever was published last (release or snapshot)

[pierre]

Totally understand, I also maintain OSS in my free time, and know that resources are at a premium.

💙 Feel free to reach out at pierre@ if your company wants to explore sponsoring options.

If/when I am able to contribute, do I need to do anything aside from sending a PR with code update + test?

That's it! 😄

it is looking like we may need to investigate the nuances of Github's packages a bit more

Yeah, I vaguely remember that GitHub support was hard to implement... Let us know if you can engage GitHub support (if you have access to it at your company).