RFC: fail fast on Travis for old queued builds in same PR or branch

[tkelman]

As suggested by @yuyichao in #12292 (comment)
We're doing this on AppVeyor already, and lately the queue has been just about as bad on Travis thanks to now having 2 appveyor workers that aren't freezing and timing out any more.

I'll push a few dummy commits to this branch to test whether or not this works properly.

[tkelman]

Alright, pretty sure this is working correctly for PR builds now. It doesn't help for "push" builds though, where Travis builds the branch for committers filing PR's from branches within this repo. We could extend the setup to check for branch builds too, just whitelist master and release branches since it's good to build every commit for those by default. Thoughts?

[yuyichao]

Go for it if it's easy to implement? FWIW, personal travis is very easy to set up and use simultaneously with the JuliaLang one if more tests are needed.

I see that the url is hard coded in the script. Would be nice to make sure that it doesn't give an error on personal travis.

[tkelman]

I'll see if there's a Travis env var or two for the JuliaLang/julia part.

Branch builds is more code that I haven't written yet, but shouldn't be too bad.

[tkelman]

Hm, I don't actually see branch name anywhere in the Travis API: http://docs.travis-ci.com/api/#builds

Seems like a pretty big omission, maybe I can ask for them to add it.

[yuyichao]

git symbolic-ref --short HEAD ?

[tkelman]

No, I can use $TRAVIS_BRANCH to get what branch the worker is on, what I want is to query the API to find the latest push build for a given branch. Turns out you don't go through the builds endpoint for that, you go through branches http://docs.travis-ci.com/api/#branches. So it should work, I think. About to test.

[yuyichao]

What is TRAVIS_BRANCH set to for PR? Especially the ones that comes from a different repo.?

[tkelman]

It's the branch targeted by the PR http://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables

So I may want to put this case block inside an else so it doesn't apply for any PR's.

[yuyichao]

or just return 0 early for pr

[yuyichao]

Looks good. Let's do it when the CI turns green.

[tkelman]

No harm in trying it for a while and seeing if we like it, whether it helps, or whether my scripting is horribly buggy. Revert if it causes any issues, accidentally cancels any false-positives, etc.

[yuyichao]

Should we merge this?

[StefanKarpinski]

Seems like a reasonable change. In general, we could optimize time-to-failure by prioritizing tests better.

[tkelman]

Not sure what you mean by prioritizing tests. Going through the timings and rearranging the order?

[timholy]

This is great, very glad to have it. Thanks, @tkelman!

[StefanKarpinski]

By prioritizing, I meant front-loading the ones that are most likely to fail. Of course, that's hard to know.

[StefanKarpinski]

Possibly clever idea: run test files corresponding to files that changed in a commit first. Requires knowing what has changed in a PR and having a way of mapping files in base/ to files in test/. If we were more consistent with naming, then the mapping would be easy.

[kshyatt]

I like the idea of running the tests in "danger" order. We should rename the files in test/ anyway - finding the tests in linalg1 or linalg3 is a giant pain.

[KristofferC]

This only works on the Mac builder?

For example, here the build stopped due to "There are newer queued builds for this pull request, failing early." but only for the Mac builder:

[yuyichao]

The check is at the very beginning of the build so if you push a commit after that, the fail fast will not trigger. The mac builder is generally the slowest one and that's probably why it is the only one that triggers in this case.