buildbot, or Continuous Integration

[staticfloat]

I figured I'd start an issue on this rather than necro the email thread, but I'm interested in trying to setup a buildbot, primarily to get as close to immediate feedback on breaking changes as possible, with the secondary objective of turning those builds into binary releases eventually.

I have little experience with anything outside of OSX and Linux, so I'm mostly looking to start discussion again and get the ball rolling. Last I read, one of the choke points was finding an easy-to-use software package to actually manage this kind of task. Although a shell script running git clone ... && make testall, and sending the exit status to a webpage somewhere would work, are there killer features that kind of approach lacks? Is it overengineering to ask for more? (Ideally we'd have something ridiculously awesome like speed.pypy.org, tracking not only build progress but performance benchmarks, etc....)

[ViralBShah]

@loladiro is exploring this. He can say more based on his progress.

[Keno]

Yeah, I wrote something that should work nicely (@julia-ci) , but I need to figure out hosting.

[staticfloat]

I took a hack at this to be used with travis-ci, and it looks like it works!

The current situation is a little hacked-up because due to the lack of cutting-edge Ubuntu packages available, each time we commit we contribute to global warming by recompiling SuiteSparse, zlib, and FFTW, so compile times are >10 minutes. On the other hand, it was pretty darn easy to setup, and as soon as we get an Ubuntu PPA up and running, the compile times should drop down to the 2-3 minute range. Oh, it runs the testsuite as well. (Note that the zlib test is currently failing)

[ViralBShah]

Why not just use the libraries that Debian provides? I think that we do not use the cutting-edge features in any of these libraries.

[Keno]

I looked into Travis when I first looked at this, but the problem is that Travis is Ubuntu-only (i.e. no Windows/FreeBSD/OS X), that it doesn't allow you to get binaries back out and that they cap compilation at 15minutes (this isn't as bad as the others, but we can't do a whole compilation run). I also remember talking to the guys at Travis what to do about that, but they said that Travis probably isn't suited too well for a project like Julia. That being said, what could work is setting up own version of Travis (it's open source after all, and has evolved a lot since I last looked at it) and use that. If I had more time, I'd be all over this, but I won't be able to look at it for another month and a half or so (not to mention all the windows stuff that's still to be done).

[staticfloat]

@ViralBShah: I'll continue that discussion in the Ubuntu PPA thread. The long and the short of it is that it doesn't "just work", although some of that could be because of me, not Julia.

@loladiro: All valid points. The compile time limit hasn't been a problem for me except when I just tried getting Julia to install ALL of her dependencies. As long as we keep LLVM out of the picture, we have a good shot at finishing before the time limit, and if we also keep BLAS/LAPACK out, there's no problem at all. I'm not sure that we need to test compilation of dependencies on a continuous integration server, I'd say it's better to have it only test the compilation and testsuite of julia.

Setting up our own version of Travis could work, but it still lacks support for other versions of Linux, much less OSX/Windows support. Things look hopeful in that regard, but remain hopeful. My guess is that when Travis supports other operating systems, they will provide those through their service as well.

I was thinking only in the CI sense, not the buildbot when I was putting this together. In the ideal world, how would you want to store automatically generated binaries? Automatic, anonymous upload to an FTP server somewhere?

[staticfloat]

I've made progress on this front (Waiting for SuiteSparse and zlib .deb's to compile, then I think I'll have gotten all the important dependencies), but I've run into a weird test error. Does anyone have experience with what this error might mean? Is my llvm .deb perhaps screwed up somehow?

[nolta]

I'm seeing the same error on the mn/fasterdict branch.

[staticfloat]

Can you give me a better idea of the environment you're using? The above is being compiled on a 32-bit clean VM, (you can see exactly which packages are installed other than build-essential in the log above) so I feel we have a good shot at figuring out exactly what's causing this.

[nolta]

Fedora14 & OS X 10.8.2, both 64-bit.

[staticfloat]

Interesting. Installing (via Homebrew) on OSX 10.8 causes the same crash on nm/fasterdict. I may have to bisect your branch later today.

Just to verify: you're not seeing this on master for either of your machines? (The above error log is from my travis branch, which shouldn't be too far behind JuliaLang/master)

[nolta]

No, i don't see this error on master.

[nolta]

Correction: i don't see this error on 64-bit systems. However, i do see this error on freebsd9/32-bit. See #1479.

[staticfloat]

All tests pass again. Most of the work was in #1451 getting reliable packages for the dependencies, but the travis bot can compile julia in about 4 minutes now. I'm not a travis convert, but I think their integration with github is really nice.

Keno, I'd like to discuss what other options we can explore if Travis ends up not working out. We could also simply use Travis only for automatically testing pull requests/commits against the test suite and not build binaries on it at all, but it would be nice to kill two birds with one stone. (Forgive the violent imagery)

[ViralBShah]

Can we close this one?

[Keno]

Yeah, I still have some ideas, but it no longer warrants an issue.

[ViralBShah]

Is it possible to prevent travis builds when the modifications are to documentation files?

[Keno]

http://about.travis-ci.org/docs/user/how-to-skip-a-build/

[StefanKarpinski]

That's pretty ugly. There's no real benefit to us to skipping builds.

[ViralBShah]

Yes, this is not a good solution. I was hoping that there could be a way to avoid building when the only files changed are files with extensions such as .md, for example.