Testing and CI

[jbenet]

http://appveyor.com seems much faster than travis-ci. Could run both.

---

Update: read #283 (comment) below

[whyrusleeping]

Would also be cool to do a coverage badge with coveralls.io

https://coveralls.io/repos
https://coveralls.zendesk.com/hc/en-us/articles/201342809-Go

[btc]

+1 for code coverage

[jbenet]

Update.

Our tests take a long time

We have the following issues

1. Some are too wasteful to run every time
2. Some are timing based :(
3. Travis is really slow and painful
4. Cannot test across all archs
5. Travis doesn't have FUSE

1. Some are too wasteful to run every time

@chriscool suggested using an EXPENSIVE prereq in the sharness tests. There already are some t.Skips in the go tests. We should have an env var with which to run them (and a make test_expensive or something like that).

We could setup a box or two to run nightlies, though i'd prefer CI systems running as much of the tests as possible, as finding the issues BEFORE merging to master is much nicer than after.

2. Some are timing based :(

We should move away from this. As we build tests for more complicated pieces of the protocol, please stay away from timing. (i'm the worst offender)

3. Travis is really slow and painful

Perhaps we can try a different CI system:

• https://drone.io/ (free for open source)
• https://codeship.com/ (free for open source)
• https://semaphoreapp.com/ (free for open source)
• http://wercker.com/ (free for open source)
• https://ci.appveyor.com/project/jbenet/go-ipfs (havent set it up properly, looks complicated gross)

These all look good, not sure how to choose. Optimizing for:

• simplicity
• speed
• github integration
• tail logs
• good notifications (emails when you break + fix things, but not too spammy)

I'd like not to waste too much time on this. I'm happy to pay for it, too, with a ceiling of $50/mo.

4. Cannot test across all archs

Eventually we'll need a few boxes people can run tests in (e.g. one in each arch). We could setup a linux and bsd digital ocean droplets, and use a mac mini I just bought. I dont think these will make sense for CI, but rather for nightlies, or debugging special cases. If you'd like to set this up / need to test on different archs, happy to grant access on a case-by-case basis (just pm me on irc).

5. Travis doesn't have fuse

Same as 4.

---

All these things are important. However, i dont want to waste a ton of our time on this right now, given we have more pressing things getting the alpha to ship.

[chriscool]

1. Yeah, I will have a look at an EXPENSIVE prereq and a make test_expensive. The 100MB file test will have the EXPENSIVE prereq and I will add a new 5MB one without the prereq.

2. Are you talking about some sharness tests too?

3. It looks like https://drone.io/ has go 1.2 and there are errors with vendored packages. I can test the others.

   $ git clone git://github.com/chriscool/go-ipfs.git /home/ubuntu/src/github.com/chriscool/go-ipfs 
   Cloning into '/home/ubuntu/src/github.com/chriscool/go-ipfs'...
   $ go version
   go version go1.2 linux/amd64
   $ echo "GOPATH: $GOPATH"
   GOPATH: /home/ubuntu
   $ go get
   $ make test
   go test ./...
   blocks/blocks.go:9:2: cannot find package "github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multihash" in any of:
   /usr/local/go/src/pkg/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multihash (from $GOROOT)
   /home/ubuntu/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-multihash (from $GOPATH)
   blocks/blocks.go:10:2: cannot find package "github.com/jbenet/go-ipfs/util" in any of:
   /usr/local/go/src/pkg/github.com/jbenet/go-ipfs/util (from $GOROOT)
   /home/ubuntu/src/github.com/jbenet/go-ipfs/util (from $GOPATH)
   blocks/set/dbset.go:4:2: cannot find package "github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore" in any of:
   /usr/local/go/src/pkg/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore (from $GOROOT)
   /home/ubuntu/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore (from $GOPATH)
   

4. I can have a look at that after the above.

5. Yeah.

[chriscool]

About make test_expensive as far as sharness tests are concerned, I don't think it is so useful as you can do make test TEST_EXPENSIVE=1 for example or make test TEST_EXPENSIVE=1 TEST_NO_FUSE=1 and it will work properly.

Regarding go tests, there are a lot of:

        if testing.Short() {
                t.SkipNow()
        }

so maybe we should run go test -short ./... when TEST_EXPENSIVE is NOT set.

[jbenet]

About make test_expensive as far as sharness tests are concerned, I don't think it is so useful as you can do make test TEST_EXPENSIVE=1 for example or make test TEST_EXPENSIVE=1 TEST_NO_FUSE=1 and it will work properly.

Yeah, i meant from the project root makefile, we can have something like this:

test:
    go test -test.short ./...
    cd test/ && make test

test_expensive:
    go test ./... 
    cd test/ && make test TEST_EXPENSIVE=1

or whatever

[chriscool]

make test TEST_EXPENSIVE=1 works from the project root. If you are in the "test" directory you only need make TEST_EXPENSIVE=1.

[chriscool]

Otherwise I am ok with having make test run go test -test.short ./... and make test_expensive run go test ./....
I will send a PR.

[chriscool]

Codeship seems to work fine for our project.

It took 2min 33 seconds to build everything and pass the tests.
The tests were run using "make test TEST_NO_FUSE=1"

Here is the project (hope you can access it):

https://codeship.com/projects/48926/

[chriscool]

I could invite only jbenet as his email address is easy to find.
If you want me to invite you to the project, you can email me at: chriscool@tuxfamily.org

[btc]

It doesn't hurt to have redundancy. We now (also) have a Jenkins box.

https://build.protocol-dev.com

When code is pushed to jbenet/go-ipfs, the following tests are run in parallel:

• sharness expensive (no fuse)
• go test
• build docker image
• build all commits on branch (w/ rebase exec. thanks @chriscool)

[jbenet]

great work on this @maybebtc and @chriscool 👏