Testing Infrastructure

[jbenet]

Recap:

• Move off jenkins #9
• Testing and CI kubo#283
• Get travis to test all commits kubo#1220 (comment)

---

We should organize our testing infrastructure concerns.

Our constraints are:

• (hard) robust, reliable way to test our code
• (hard) able to run all the tests (with iptb, we no longer need docker)
• (med) smooth, friendly UX for contributors -- including new contributors (travis wins, by far).
• (hard) well integrated with github PRs and commit status indicators (travis wins by far)
• (med) no management overhead on our side (jenkins was a huge pain, that's why we got rid of it)
• (soft) tests take <5min
• (soft) cheap. But doesn't even have to be free. i keep trying to give travis-ci money (they won't take it :( )

Unfortunately, after trying multiple CI systems, always end up coming back to travis. (We used jenkins for a long time and it was a huge pain to manage. life is much better without it).

Close seconds are:

• https://circleci.com/ -- pricey but hear good things. not sure how well they integrate with OSS
• https://codeship.com/ -- they keep improving, but don't take security seriously.

For testing on windows, we could use:

• http://www.appveyor.com/

[jbenet]

Also, relevant exchanges with Travis-CI -- I should say that I'm very impressed with their support:

http://gateway.ipfs.io/ipfs/QmZeCPZPkgSRoLEHKFVGxsEJKiuGye16ekXjtyVdorjdkW/email.html

[chriscool]

@jbenet could you give more details about the problems with Jenkins?
At my current job we use Jenkins and at my previous job they used hudson (though they started switching to buildbot), and my experience is that it works fairly well and is not so painful to manage.
But maybe it depends on the setup, how the build slaves are configured or managed?

[chriscool]

Also when you say "tests take < 5min", this is for one commit.
But it could be nice if we could test a 10 commit long branch without having to use 10 branches.

And about running all the tests, does that means FUSE tests too?

[whyrusleeping]

@chriscool i think the issue with jenkins (at least in my experience) was that it was confusing and we (well at least, i) didnt quite understand how to get it all working. It wasnt as simple as we wanted

[chriscool]

@whyrusleeping maybe I can have a look at it on Saturday or Sunday.

[whyrusleeping]

@chriscool that would be really great :) I'd love to have a more full test suite running again

[jbenet]

i found jenkins unintuitive, complicated, and unpleasant to use. if someone else wants to take care of it and have it running-- that's fine, but i got tired of doing it. I don't want to have to look at it. travis works much better so far. travis is hyper clear-- everything is very simple, intuitive, or very well documented. it's also pleasant to use.

I would really, really love to use this: http://build.golang.org -- https://github.com/golang/build -- i just took a look and it doesn't seem very complicated.

On the branches, ideally we could convince travis to do it automatically for us, but https://github.com/jbenet/git-push-each is not bad. We also get the indicator per-commit, which normally wouldn't be there. the last thing needed is a bot that automates it. it's not a big deal to have extra branches that are added and removed. After all, branches are very, very cheap.

[chriscool]

At the bottom of https://github.com/golang/build there is:

If you wish to run a Go builder, please email golang-dev@googlegroups.com first.
There is documentation at https://golang.org/wiki/DashboardBuilders but
depending on the type of builder, we may want to run it ourselves, after you
prepare an environment description (resulting in a VM image) of it. See the env
directory.

I wonder if we should contact them?

[jbenet]

Yeah perhaps! I think they mean running a gobuilder to help with building go itself. They probably don't want to take on the challenge of supporting other groups, but they may at least help with pointers on how to set it up to test other things :)

—
Sent from Mailbox

On Sat, May 16, 2015 at 1:51 PM, Christian Couder
notifications@github.com wrote:

At the bottom of https://github.com/golang/build there is:

If you wish to run a Go builder, please email golang-dev@googlegroups.com first.
There is documentation at https://golang.org/wiki/DashboardBuilders but
depending on the type of builder, we may want to run it ourselves, after you
prepare an environment description (resulting in a VM image) of it. See the env
directory.

I wonder if we should contact them?

Reply to this email directly or view it on GitHub:
#20 (comment)

[chriscool]

Also can we run https://github.com/jbenet/git-push-each in another Travis job? Is there a limit in the number of Travis jobs? And what is the difference between the 2 jobs we have (pr and push)? Is it just that one is run on the PR branch and the other one on the master after the branch is merged?

[chriscool]

And about gobuilder, yeah it looks like they mean running a gobuilder to help with building go itself.
This is a bit worrying because perhaps it means that gobuilder is not well suited to build something else. We will see when we contact them.

[jbenet]

@chriscool

take a look at:

• http://docs.travis-ci.com/user/build-configuration/
• http://docs.travis-ci.com/user/speeding-up-the-build/
• http://docs.travis-ci.com/user/build-configuration/#The-Build-Matrix

seems like we could break up the "go test" and "sharness" parts with different build targets. maybe:

env:
  - TEST_SUITE=go_test
  - TEST_SUITE=sharness

script: "make $TEST_SUITE"

or something.

is there a limit in the number of Travis jobs?
probably, not clear how big. we can ask them if we hit a limit.

what is the difference between the 2 jobs we have (pr and push)? Is it just that one is run on the PR branch and the other one on the master after the branch is merged?

Correct-- the idea is to see whether the errors are on the branch or in merging with master. i think they started doing that automatically because people pointed out that was a problem.

[harlantwood]

Circle CI is awesome. It's free for open source. I use it exclusively for new ruby and node projects, including continuous deployment to Heroku and github pages. I haven't used it with docker or go. I used Travis before Circle, and I'd never go back. I can't point to features Circle has that Travis doesn't; but everything feels even cleaner, more lucid, more usable. They are always adding new features, and are eager to hear what users want.

[harlantwood]

Great to see that go-ipfs is passing now on Cirlce CI (nice work @jbenet).

I know that we've installed both Circle and Travis on a few projects. Which do people prefer? It would be great to standardize, especially to move forward on aggregating CI for app IPFS projects into a single place.

[jbenet]

@chriscool did you ever email to golang/build folks?

[chriscool]

No I didn't I can do it these days.

[chriscool]

Ok, I sent an email entitled "Running a Go builder for IPFS", but it looks like it didn't make it to the golang-dev group:

https://groups.google.com/forum/#!forum/golang-dev

[jbenet]

the groups are moderated -- takes some time for mods to let the messages through

[jbenet]

In this thread i started: https://groups.google.com/forum/#!topic/golang-nuts/kRq8Ow4qEeU bradfitz says this is pretty doable. I think we should do it. cc @whyrusleeping

[chriscool]

I don't know much about it but I am ok with working on this.

[ghost]

Closing in favor of #100