Publish jfrog-cli-go binaries to Chocolatey for Windows and Apt/Yum for linux

[solvingj]

I really want to use the jfrog cli both on my machine, and in my CI processes, so I'd like to help your team get the binaries published to the package managers for Windows and Linux. This used to be a very tedious process, but there is actually a shortcut now.

There is an amazing collection of tools for unifying and streamlining cross-platform packaging for any kind of binaries, which also happens to be written in Go. All the tools are "detailed" in the following repository, although his process isn't quite what you should follow.

https://github.com/mh-cbon/go-github-release

If this team would be willing to take on the burden of publishing to Chocolatey, RPM, and DEB repositories, I will submit some PR's with instructions to get it started. If not, then I won't bother.

[eyalbe4]

@solvingj,
Do you think that the process of publishing JFrog CLI with these package managers can be automated?

[solvingj]

Yes, they can be fully automated via CI.

[eyalbe4]

@solvingj,
It would be great if we could collaborate!
We'd gladly accept your contributions through pull requests.
Let's communicate here and agree on the solution.

[solvingj]

Worked on it a bit this weekend. I hope to submit a PR soon.

[solvingj]

So much for soon. Haven't had time to look in past month. Will take another look tonight/tomorrow.

[eyalbe4]

If you get a chance to look into this @solvingj, it best to use the dev branch.

[solvingj]

PR submitted, have a great weekend!

[solvingj]

HI @eyalbe4 , has anyone had a chance to start looking into this?

[eyalbe4]

Not yet @solvingj, but we'll get it soon :)

[jgeorgeson]

@eyalbe4 I noticed on https://jfrog.com/getcli/ there are APT/Yum repos now. I'm the maintainer of the Chocolatey package, https://github.com/jgeorgeson/chocolatey-packages, I'd be happy to donate it in order to have it be official and included in your release process.

[eyalbe4]

That would be great @jgeorgeson! Thanks!
We currently using the Jenkinsfile located at the root of this repository to build all the supported package managers.

[jgeorgeson]

I assume that since you're using AppVeyor to build the Windows executables, you don't have any Windows agents for either your Jenkins or jFrog Pipelines (either thick agents or containers). I don't think will be possible to build the Chocolatey package with PowerShell Core on non-Windows platform. Neither Chocolatey nor the recommended automatic updator packaging framework support PowerShell Core.

• https://chocolatey.org/docs/installation#requirements
• [Feature Request] Add PowerShell Core Support majkinetor/au#158

Fundamentally, Chocolatey packages are NuGet packages with extra meta-data, so it might be possible to package and publish using nuget.ext with Mono in a Linux container, but I've never attempted that.

So would it be ok to put the Chocolatey packaging in the AppVeyor build, or is working it into Jenkinsfile a hard requirement?

[eyalbe4]

@jgeorgeson,
Actually, we use AppVeyor only for running the teats on Windows. We build all the JFrog CLI executables, including the Windows executable on a Linux Jenkins agent, using the Jenkinsfile included in this repo.
I wouldn't say running the build on a Linux agent is a hard requirement, but having the entire release process running on one agent does make things easier to maintain.
Can we try using nuget with and mono?
Since the existing Jenkinsfile already takes care of the heavy lifting of creating the windows executable and signing it (see the code in the Jenkinsfile), I think it may be possible to use another docker image with nuget and mono, which will take the signed executable and package it.
What do you think?

[jgeorgeson]

Ok, will see if I can make that work.

[jgeorgeson]

So it turns out there's already interest and traction getting Chocolatey to build and run under mono, including some Docker image work already done.

• Dockerfile to Enable Mono Based Choco Builds chocolatey/choco#1153
• https://github.com/chocolatey/choco/tree/master/docker

And the contributors of those efforts publish an unofficial Linux Mono Docker image

• Publish a Docker Image using Docker Hub chocolatey/choco#1718
• https://github.com/Linuturk/mono-choco

I played around with it in WSL and am able to install the resulting nupkg with Chocolatey, so it's looking to good to integrate into your Jenkins release pipeline. It might not be able to pick this back up again until the weekend, but you can get started on becoming package maintainer here. You'll need to setup an api key to use in Jenkins for the choco push to publish it.

[jgeorgeson]

Hoping to have a PR opened today.

Looking at the Jenkinsfile, it appears that the executable is built separately for the unpackaged publish to Bintray vs the packaged (RPM/DEB/NPM) versions. It seems the checksums are different between these three distributions

https://bintray.com/jfrog/jfrog-cli-go/jfrog-cli-linux-amd64/1.39.5
https://releases.jfrog.io/artifactory/jfrog-debs/pool/jfrog-cli/jfrog-cli-1.39.5.x86_64.deb
https://releases.jfrog.io/artifactory/jfrog-rpms/jfrog-cli/jfrog-cli-1.39.5.x86_64.rpm

The current Chocolatey package I publish scrapes the checksum from Bintray, and just runs PowerShell script to download the exe from Bintray and verify the checksum. For official packaging though, it's better to bundle the application, especially for a lightweight standalone executable like we have here. In this model it's required to have a VERIFICATION.txt file included

https://github.com/chocolatey/package-validator/wiki/VerificationFileMissing

Since I can't suggest using the checksum from Bintray as the verification, I tried to use signtool to verify signature, but that says the root cert isn't trusted.

> & 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64\signtool.exe' verify .\jfrog.exe
File: .\jfrog.exe
Index  Algorithm  Timestamp
========================================
SignTool Error: A certificate chain processed, but terminated in a root
        certificate which is not trusted by the trust provider.

Number of errors: 1

I can document the certificate chain that is used to sign the exe though, hopefully that will be adequate.

[jgeorgeson]

@eyalbe4 @RobiNino If you provide me the username(s) registered on Chocolatey.org I can add as package maintainers there directly.

[eyalbe4]

@jgeorgeson,
The username on Chocolatey is jfrog.

[jgeorgeson]

Added, it says pending approval, not sure if that goes to the site admins or you.

[eyalbe4]

@jgeorgeson - I have just accepted the request to become the package maintainer.

[eyalbe4]

@jgeorgeson,
We tried to publish patch 1.39.6 to Chocolatey for the first time time today. The following docker run command failed on the Jenkinsfile:

docker run -v \$PWD:/work -w /work $architecture.chocoImage push --apiKey \$CHOCO_API_KEY jfrog-cli.$version.nupkg

So it looks like the package creation was successful, but the publication failed. We used the API key we received through the Chocolatey website.
This is the error in from the Jenkins build log:

16:54:54  Attempting to build package from 'jfrog-cli.nuspec'.
16:54:59  Successfully created package '/work/jfrog-cli.1.39.6.nupkg'
16:54:59  [Pipeline] }
16:54:59  [Pipeline] // withCredentials
16:54:59  [Pipeline] }
16:54:59  [Pipeline] // dir
16:54:59  [Pipeline] }
16:54:59  [Pipeline] // stage
16:54:59  [Pipeline] }
16:54:59  [Pipeline] // dir
16:54:59  [Pipeline] }
16:54:59  [Pipeline] // node
16:54:59  [Pipeline] End of Pipeline
16:54:59  [Checks API] No suitable checks publisher found.
16:54:59  org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such field found: field java.lang.String nupkg
16:54:59  	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.unclassifiedField(SandboxInterceptor.java:425)
16:54:59  	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:409)
16:54:59  	at org.kohsuke.groovy.sandbox.impl.Checker$7.call(Checker.java:353)
16:54:59  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:357)
16:54:59  	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
16:54:59  	at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
16:54:59  	at WorkflowScript.publishChocoPackage(WorkflowScript:213)
16:54:59  	at ___cps.transform___(Native Method)
16:54:59  	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
16:54:59  	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
16:54:59  	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
16:54:59  	at jdk.internal.reflect.GeneratedMethodAccessor330.invoke(Unknown Source)
16:54:59  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:54:59  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
16:54:59  	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
16:54:59  	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
16:54:59  	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
16:54:59  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
16:54:59  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
16:54:59  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
16:54:59  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
16:54:59  	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
16:54:59  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
16:54:59  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
16:54:59  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
16:54:59  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
16:54:59  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
16:54:59  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
16:54:59  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
16:54:59  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
16:54:59  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
16:54:59  	at java.base/java.lang.Thread.run(Thread.java:834)
16:54:59  Finished: FAILURE

Any idea what causes this?

[jgeorgeson]

I think it's interpolation of jfrog-cli.$version.nupkg not working as intended. It's trying to find nupkg as a field of string object $version. Not sure how to tell Groovy to stop interpolating at the . separating the two (like using {} for she'll env variables).

[jgeorgeson]

Looking at https://groovy-lang.org/syntax.html#_triple_single_quoted_string I think jfrog-cli.${version}.nupkg will work.

[eyalbe4]

Indeed @jgeorgeson - #857 fixes the issue. Thanks!

[jgeorgeson]

I think it's safe to close this issue by now! Also, please remove me as a package maintainer over here.

[eyalbe4]

Done @jgeorgeson.
Thanks so much for creating this package. We truly appreciate it!