CI target architectures for libuv

[rvagg]

Continued from libuv/libuv#12 also see #1 for additional context.

Here's a strawman proposal for architectures libuv should be tested against. They are split in to 3 classes mainly based on how difficult they will be to set up and include in the build set and how important they are to have solidly tested builds against.

---

Class A

• CentOS 6 64-bit (EL6)
• CentOS 6 32-bit (EL6)
• CentOS 7 64-bit (EL7)
• Ubuntu 10.04 LTS (Lucid Lynx) 64-bit
• Ubuntu 10.04 LTS (Lucid Lynx) 32-bit
• Ubuntu 12.04 LTS (Precise Pangolin) 64-bit
• Ubuntu 12.04 LTS (Precise Pangolin) 32-bit
• Ubuntu 14.04 LTS (Trusty Tahr) 64-bit
• Ubuntu 14.04 LTS (Trusty Tahr) 32-bit
• Ubuntu 14.10 (Utopic Unicorn) 64-bit
• Debian stable (wheezy) 64-bit
• Debian stable (wheezy) 32-bit
• Windows Server 2008 R2 + Visual C++ 2012 64-bit
• Windows Server 2008 R2 + Visual C++ 2012 32-bit
• Windows Server 2012 R2 + Visual C++ 2013 64-bit
• Windows Server 2012 R2 + Visual C++ 2013 32-bit
• Mac OS X 10.8 (Mountain Lion) + XCode 5
• Mac OS X 10.9 (Mavericks) + XCode 5
• Mac OS X 10.10 (Yosemite) + XCode 6

Class B

• CentOS 5 64-bit (EL5)
• CentOS 5 32-bit (EL5)
• SmartOS
• ARMv6 32-bit (Linux)
• ARMv7 32-bit (Linux)
• ARMv8 32-bit (Linux, one day, when suitable hardware & OS is available)
• ARMv8 64-bit (Linux, one day, when suitable hardware & OS is available)
• FreeBSD stable/9 (maybe)
• FreeBSD stable/10

Class C

• MinGW 32-bit
• MinGW 64-bit
• POWER8

---

The open questions, for me at least, are:

• What build configurations are acceptable here? Is it good enough to build debug and test that in all cases or do we need some testing of release builds somewhere?
• Windows support: what is expected of libuv by its various consumers? We are being pushed in to MSVC2013 territory for Node.js because of the upstream adoption of C++11 by V8 but this is obviously not the case for libuv, so how much depth is needed there and are the MinGW builds still important (initially proposed in CI build platforms #1).

Are there any concerns here held by the libuv team? / @indutny @saghul @bnoordhuis @piscisaureus (I'm guessing here at who constitutes libuv-core btw).

[tkelman]

I'll second what @saghul said in #1 (comment) about MinGW. The requirements of Julia, a downstream consumer of libuv (currently on a fork but would like to fix that), cannot be met by MSVC (even 2013; 2015 should help some but not completely). Legacy 32-bit-only mingw.org is far less important to us than the MinGW-w64 toolchain which has both i686 and x86_64 variants, but issues do still occasionally get filed at libuv when something breaks with the legacy toolchain.

[rvagg]

Moved "Mac OS X 10.10 (Yosemite) + XCode 6" up to class A, thanks @SomeoneWeird for picking that up. Does it still make sense to include 10.8 in the list here?

A couple of additional items worth noting:

This list of full architectures will be reserved for use by the libuv-core team rather than for everyone submitting pull requests. The list of people who's pull requests will trigger a full-set build will be controlled by that core team and will be kept relatively small to a trusted set due to security concerns of running arbitrary code from pull requests on most of these platforms. At the moment we have a containierized build set working @ http://jenkins.node-forward.nodesource.com/ that will be used to test incoming pull requests from untrusted sources. Core members will then be able to request a full build of any pull request if it makes sense to do so (the exact mechanism of this request is yet to be decided). The containerized set simply includes the last 3 LTS releases of Ubuntu but we could expand that if necessary and possibly even include other architectures where this is easy (SmartOS/Solaris, FreeBSD). So that set is a matter for another discussion but will also come down to difficulty involved in coordinating this to make it workable.

The other item worth noting here is that we have hardware contributions from multiple companies to make this happen: DigitalOcean (Linux), Rackspace (Linux + Windows), IBM (via Softlayer) (Linux + Windows + POWER), and Voxer have also stepped up with a couple of Mac Minis in their datacenter. We also have people power provided by NodeSource (myself and @ryanstevens) and a large number of very interested parties who want to help out because they care deeply about the success of Node.js and libuv and believe this is the best place they can contribute. Over the coming months we'll be trying to make this project much more organised and give clear avenues for additional contributions by companies and community members.

[jbergstroem]

Does these classes also reflect usage? Will all classes have buildbots? Are fails accepted in Class C? I think a better definition of these classes would improve understanding of what the greater goal is.

Since hardware/vm is pretty cheap, I think the Linux prevalence should be diversified with BSD's (Open, Net, Dragonfly comes to mind). Illumos/SmartOS also feels like a pretty relevant inclusion. Since the build isn't limited by V8 (which would be the showstopper for most of nodejs), libuv would only improve by testing on these os'es or even additional architectures.

Also; above targets seems to be "point in time" distributions (10 lts vs 12 lts) -- a possibly more relevant mindset would be testing functionality such as disabling ipv6 or similar (ref libuv/libuv#10). Point being that the major difference between all those Linux'es being toolchains or kernel versions.

[tkelman]

Oh, and I should note that I very frequently cross-compile libuv (and lots of other things) from Linux to Windows via MinGW-w64 and autotools. As far as libuv's build system goes I'd like to see that continue to work, though other communities might not do that kind of thing as often. Cross-compiling is highly relevant for ARM as well, but I haven't seen any mentions of it yet.

[bnoordhuis]

Does it still make sense to include 10.8 in the list here?

Data point: I still use 10.8. I may very well be an outlier, though.

I think the Linux prevalence should be diversified with BSD's (Open, Net, Dragonfly comes to mind). Illumos/SmartOS also feels like a pretty relevant inclusion.

I don't really disagree but the number of BSD users that are not FreeBSD users is practically a rounding error, and I say that as the one who did the DragonFly BSD port and spent the most time maintaining the OpenBSD port. It's okay to add such platforms to the CI matrix but test failures (and arguably build failures too) should not block tier 1 platforms.

Likewise with Illumos/SmartOS. Realistically, the only users are Joyent customers and as they pay Joyent good money, I think it's fair that responsibility for maintaining the Illumos port falls to Joyent (or their customers, if they are willing to step up.)

[rmg]

My opinion on platform support, without any real background in the libuv project

I think it would make sense to treat all of the listed platforms as class A and only move them to class B if/when they start failing in a way that puts a drag on productivity. There is no class C in this scheme because it's a wish list and support for those platforms belongs in an issue tracker instead.

[jbergstroem]

@bnoordhuis Without going into the guesttimation game for user/os adaption, my suggestions didn't assume a successful build or test suite - rather the fact that if hardware is available the effort to increase visibility is pretty low. Portability is generally a good thing and exposing shortcomings doesn't mean that it has to get highest priority. This is also why my comment started with "what does different classes mean".

[rvagg]

Regarding VMs being "cheap" leading to wanting greater diversity, the equation isn't as simple as throwing as many options as we can think of against this project.

• The maintenance overhead of a large diverse cluster is quite large. It's manageable with enough input from the community and if people concerned with libuv want to get involved in helping maintain an active cluster then that's great. I'll have to monitor the situation over time and if the burden ends up falling on me (or someone else, or a small group) then we'll have to trim to make it manageable.
• Thankfully libuv is a quick build and test but we still need VMs that are adequately performant so as not to have slow feedback cycles so we can't just throw super-cheap VMs at this. We have very generous support by some hardware providers at the moment but we do need to be careful to manage the burden that we're placing on them.
  I suggest we target the obvious choices now and expand over time as it becomes obvious that expansion is needed—just like you would add a test case when you encounter a regression, let's expand when it becomes clear that we have an important blind-spot.

What the "obvious" choices are is actually a matter for the libuv core team and I'd like them to come up with that list after taking into account the discussion here. That includes architectures, build modes etc.

Regarding cross-compiling: we can test that it's possible to do the compiling but testing the builds produced from cross-compiling is a non-trivial exercise and would involve coordination of multiple VMs. I'm personally skeptical of the value of including cross-compiling in the CI set but I'm happy to be told otherwise by the libuv team.

Regarding the classes— I grabbed them off the README which includes details about what happens when failures occur on the lower classes. However, this is just a starting point and that list is mainly focused on Node. I expect to land somewhere different with libuv and would love to hear more discussion here on how to define the classes, how to handle failure cases in the classes, etc.

[jbergstroem]

@rvagg After reading the README it makes more sense, thanks. If there's anything I can do to help wrt build environment or maintenance, ping me at IRC (freenode, same username).

[ghostbar]

FYI, Debian wheezy is soon to be oldstable. Debian is currently on "freeze" to move current testing to stable.

[bnoordhuis]

Closing, I think we've implemented most of what is discussed here. Testing with older compilers is maybe the exception but that hasn't been a problem so far. I'll file a separate issue for that when it becomes a problem.