Cross-Platform OS Packaging and Publishing to Bintray using Travis & Appveyor

[solvingj]

Introduction

This pull request is in response to #70 . It contains a LOT of new information, and it will take time and emails or slack chats to digest it all. If you want to create a separate branch for this, I'm happy to resubmit.

Packaging

Cross-platform OS packaging with full support for the built-in OS package managers is inherently complicated. Fortunately, one OSS developer went to great lengths to bring it within reach of mortals. His Github handle is mh-cbon, and he created all the tools used for this process.

At a high level, he defined a fairly standardized JSON schema for each of the three most popular (and wildly different) OS package managers. He then applied a standardized CLI experience and created corresponding package utilities: go-msi, go-bin-deb, and go-bin-rpm. The tools read the configs, translate them to the proprietary formats, and then automates the calls to the proprietary package tools behind the scenes. It's a remarkable achievement. The best primer is to read about the tools and process detailed here (although our strategy diverges in a few areas):

• https://github.com/mh-cbon/go-github-release

My contribution was to streamline the use of his tools into effective "recipe templates" for publishing his packaging tools to Bintray rather than gh-pages and gh-release. This led me to finding and using jfrog-go-cli for the publishing. I then discovered that jfrog-cli-go could benefit from the same workflow, so I ported the templates to here. It was actually "easy" to port the recipe's from the other repos, which is a testament to the modularity and portability of the strategy. However, please note, it took actual work weeks to figure the whole strategy out, so "grokking" it all will not be trivial to learn (but at least now it's approachable).

Outputs

To see the finished products of the additions made in this PR, you can look at these repositories:

• https://bintray.com/bincrafters/public-choco
• https://bintray.com/bincrafters/public-msi
• https://bintray.com/bincrafters/public-deb
• https://bintray.com/bincrafters/public-rpm

You can intall the CLI from any of these repos using the corresponding package managers.

First you add the repo to your local list:

• Debian
  echo "deb https://dl.bintray.com/bincrafters/public-deb unstable main" | sudo tee -a /etc/apt/sources.list && sudo apt-get update

• Redhat/Centos/Fedora
  wget https://bintray.com/bincrafters/public-rpm/rpm -O bintray-bincrafters-public-rpm.repo && sudo mv bintray-bincrafters-public-rpm.repo /etc/yum.repos.d/ && sudo yum update

• Windows
  choco source add -n=bincrafters -s="https://api.bintray.com/nuget/bincrafters/public-choco"

Then install:

• Debian
  apt-get install jfrog-cli-go

• Redhat/Centos/Fedora
  yum install jfrog-cli-go

• Windows
  choco install jfrog-cli-go

Moving Forward

In theory, JFrog should have (or create) a similar set of public repos for developers to release official JFrog software through (such as jfrog-cli-go). It's kind of madness not to drink your own kool-aid in this case. So, one reasonable approach to "take over" the recipe's here is to create your own repos and CI accounts, and change all the relevant variables. You could also try to migrate them to Jenkins and go a different appraoch. I'm happy to help either way.

Also, you may choose to publish jfrog-cli-go to Chocolatey. I certainly encourage it, and it's very easy once you've published to your own nuget repository once. You may also want to submit to "apt" and "yum" central repositories, I have never done this, but would be interested in seeing how it goes. I hear it's much more involved than Chocolatey.

Outstanding "Nuances"

• Bintray Web API still has a bug, and won't let you overwrite an existing NUGET package file
  -- Thus, after the first successful build/publish of .NUGET to bintray, subsequent builds will fail
  -- I have opened two tickets for this, one to change JFrog CLI behavior , one to fix the web API
• Nuget (and therefor chocolatey) wants both i386 and amd64 packaged together.
  -- This doesn't fit well with Appveyor Matrix, which wants to build each separately
  -- You may want to modify the build step to compile both in one build, and then package both.
  -- Currently, appveyor creates both MSI, but only creates i386 NUGET package for chocolatey.
• .deb requires "i386" instead of "386", and it's more common, that's why OSARCH && GOARCH
• MAC, there is currently no related solution for MAC packaging or deployment using BREW. Future?
• Travis runs debian, can't build .rpm on debian, we use docker/fedora image on travis to create .rpm.

Good luck, and feel free to reach out to me via email for extended discussion.

[eyalbe4]

Thanks a lot for the great job and detailed information @solvingj!
Since JFrog CLI is currently built using Jenkins, we'll probably need embed this into its Jenkinsfile.
As for a MAC solution, JFrog CLI is already available using brew install jfrog-cli-go.

[solvingj]

I'm glad you think it looks good.

As for CI, I saw the jenkins file, but wanted to demo a fully working pipeline. Also, there was some possibility that your team might have wanted to see Appveyor/Travis together and just didn't have time. Feel free to use as a reference for the relevant steps.

As for brew, I meant to say that we do not currently have any packaging tool for brew in the toolset. I would like to someday see an additional tool called go-brew or go-bin-brew that mimics the others, but that would require a MAC to develop, along with some knowledge of brew. Since your team clearly has some experience in publishing to brew, it would be great if your team could somehow find time to help us get started with that.

[solvingj]

Hi Eyal, Do you work at the Sunnyvale office? Regards, Jerry

…

Sent from my iPhone

On Sep 3, 2017, at 8:07 AM, Eyal Ben Moshe ***@***.***> wrote: Thanks a lot for the great job and detailed information @solvingj! Since JFrog CLI is currently built using Jenkins, we'll probably need embed this into its Jenkinsfile. As for a MAC solution, JFrog CLI is already available using brew install jfrog-cli-go. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[eyalbe4]

Hi @solvingj
No, I work at the Netanya office.
I hope we get a chance to meet face to face in the future though :) Maybe at swampUp.

[eyalbe4]

@solvingj,
We haven't managed to push this forward.
If you are still available to help with this, I suggest we split the effort and start with one of package managers, by adding an independent build script.
I'm closing this PR for now. We can open a new one if we decide push this initiative forward.
Thanks!