Publish "unstable" build on each push to master?

[mattsooknah]

(related to #115)

There's been some interest in having an "unstable" jar published to Maven on each push to master, in addition to the "stable" release every ~2 weeks.

This would be useful for larger projects that depend on HTSJDK where a few members of the project want to contribute features to HTSJDK, without everyone having to worry about syncing their local versions of HTSJDK before building the other project.

I'm not sure on the best way to do this. We'd also need to make it very clear that it's for experimental use only. In the short term, a fork might be an easier solution, but let's keep this on the radar.

@lbergelson @akiezun @droazen are interested in this.

[lbergelson]

I am indeed interested in this. I haven't gotten a chance to look into it yet much though.

[nh13]

Have a suggestion when I get back in next week. Thanks for tracking this!

N

Thumb typed for added typos

On Feb 20, 2015, at 11:04 AM, Louis Bergelson notifications@github.com wrote:

I am indeed interested in this. I haven't gotten a chance to look into it yet much though.

—
Reply to this email directly or view it on GitHub.

[lbergelson]

@nh13 What was your suggestion for this?

[nh13]

Responded in person, posting for posterity.

[vdauwera]

I am also interested in this. What needs to happen to make this happen soon?

[kshakir]

I'll touch base on Monday. Setup is on the same order of magnitude as the existing Maven Central releases. One can publish pre-releases and/or snapshots. As htsjdk is foss, the repo can be sonatype, or use our artifactory.broadinstitute.org. Search "artifactory" on the dsde wiki for a little more info for now.

[vdauwera]

Looking fwd to this happening! We are in need of this right now for the issue referenced above.

[droazen]

Any word on this?

[kshakir]

I pinged GG last week with the following:

I have more info on publishing HTSJDK _pre-_releases. Happy to chat when you have time, and with more folks if you want to schedule a chat of involved parties. I assume because HTSJDK is OSS, you'll keep using the same host Sonatype to host these new jars. Next steps: we'll need to know 1) Who / what username currently pushes releases (assuming they will keep using their account) and 2) What versioning you want for the pre-release jars: -SNAPSHOT always downloading the latest, or git labeled versions where the consumer has to explicitly lookup the latest

Once the above is resolved, that person just needs to point the build at the right -SNAPSHOT or release repo: http://central.sonatype.org/pages/ossrh-guide.html#accessing-repositories

[kshakir]

By the way, if that mystery publisher hasn't already setup a shared credential that can be put into travis/jenkins/etc., it should be possible to create one with a few steps.

Here are a few example links I came across that touched on somewhat similar setups:

• https://gist.github.com/neothemachine/4060735
• http://benlimmer.com/2014/01/04/automatically-publish-to-sonatype-with-gradle-and-travis-ci/
• Can we get Travis to deploy SNAPSHOTs to Sonatype? ReactiveX/RxJava#2586
• https://issues.sonatype.org/browse/OSSRH-7593
• http://www.scala-sbt.org/0.13/docs/Publishing.html

If interested parties -- and the person(s) who owns htsjdk publishing -- want to throw something in my calendar, happy to discuss more.

[vdauwera]

Oops, sorry for the radio silence, that's my bad. To answer the questions, I don't know who currently pushes releases but I thought it might be @jacarey; and I assume we would want the -SNAPSHOT versioning.

[lbergelson]

I'm strongly in support of individual versions per commit.

[vdauwera]

I happily defer to the expertise of my learned colleague

[kshakir]

Btw, click here for the sbt based code snippet for generating a git based pre-release version suffix per commit. Similar exists for Maven[1][2], etc. (Thanks again DR!)

[lbergelson]

@kshakir thanks for the pointer to the DSDE wiki. It turns out it's rather easy to publish to the artifactory thanks to your tutorial.

I think snapshots would be OK if they were per branch. I think that ideally, builds should be entirely repeatable, and snapshots break that assumption.

[kshakir]

So I was mistaken about Sonatype hosting pre-releases that one could take down after release.

I thought there was an intermediary step between the OSSRH repo, and the HTSJDK pre-release being pushed to Central. It turns out the intermediary step is before the artifact reaches the OSSRH release repo.

After a successful deployment to OSSRH your components are stored in a separate, temporary repository, that is private to your projects members.

-- http://central.sonatype.org/pages/releasing-the-deployment.html

Repository URL for release download access, this repository is synced to the Central Repository ...
• https://oss.sonatype.org/content/repositories/releases/

-- http://central.sonatype.org/pages/ossrh-guide.html#accessing-repositories

I'm pretty sure we want the ability to take down these pre-releases. So unless there are any objections, I'll ping @mmonnar via a help ticket for next steps towards hosting just the pre-releases on our publicly accessible, but internally maintained repo. The plan always was for pre-release consumers to add a new repo to their builds, but instead of OSSRH it'll be https://artifactory.broadinstitute.org.

[kshakir]

Credentials are now available as environment variables for "local" builds, aka not in pull requests.

The next steps to be done are wiring a script (sbt, bash, or otherwise) to use these environment variables, and publish the artifacts on each successful build if the ${TRAVIS_BRANCH} is master.

I'm juggling a number of other tasks at the moment, so please continue to ping, or even better google away and submit a patch for review.

FYI the variables one will need are for configuring sbt/ivy credentials are:

• host: artifactory.broadinstitute.org
• realm: Artifactory Realm
• username: ${ARTIFACTORY_USERNAME} (from environment)
• password: ${ARTIFACTORY_PASSWORD} (from environment)

The publish endpoints are:

• (Pre-)Releases: https://artifactory.broadinstitute.org/artifactory/libs-release-local
• Snapshots: https://artifactory.broadinstitute.org/artifactory/libs-snapshot-local

Btw, upon further thought, I'm thinking perhaps to make it unambiguous to users, htsjdk should use both git and snapshot versioning: <htsjdk.next>-<git.rev.list>-<git.rev.hash>-SNAPSHOT, for example 1.111-123-abcd456-SNAPSHOT. It's both unique, and leaves zero doubt that the version is temporary.

[ldgauthier]

@kshakir are you going to have any time in the immediate future to put into this? There's a GATK issue I was going to assign to one of our assoc. comp bios that needs an htsjdk update and it would be a little more approachable if it didn't need a merge to htsjdk master in the middle.

[kshakir]

First build is up, and should work for maven, sbt, gradle, ivy, leiningen, et al.

https://artifactory.broadinstitute.org/artifactory/libs-snapshot/com/github/samtools/htsjdk/1.137-bf78837-SNAPSHOT/

I believe this issue is complete.

[jacarey]

And our first 1.138 snapshot too! Looks good @kshakir

[vdauwera]

Thanks @kshakir!