Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilers are too old on many CI machines #1970

Closed
targos opened this issue Oct 18, 2019 · 7 comments
Closed

Compilers are too old on many CI machines #1970

targos opened this issue Oct 18, 2019 · 7 comments

Comments

@targos
Copy link
Member

targos commented Oct 18, 2019

Found with nodejs/node#30028

Expected: g++ 6.3.0 or clang++ 8.0.0

V8 requires C++14 support, meaning we need at least g++ 5.x everywhere.

node-test-commit-v8-linux

  • infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)
  • test-nearform_intel-ubuntu1604-x64-1: (CXX=g++, 5.4.0)

node-test-commit-freebsd

  • test-digitalocean-freebsd11-x64-1: (CXX=ccache c++, 6.0.0)

node-test-commit-custom-suites-freestyle

  • test-rackspace-ubuntu1604-x64-1: (CXX=ccache g++, 5.4.0)

node-test-commit-arm

  • test-packetnet-ubuntu1604-arm64-2: (CXX=ccache g++, 5.4.0)

node-test-commit-linux-containered

  • test-joyent-ubuntu1804_sharedlibs_container-x64-4: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-10: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-1: (CXX=ccache g++, 5.4.0)
  • test-softlayer-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-7: (CXX=ccache g++, 5.4.0)

node-test-commit-linux

  • test-rackspace-debian8-x64-1: (CXX=ccache g++, 4.9.2)
  • test-rackspace-ubuntu1604-x64-2: (CXX=ccache g++, 5.4.0)
@rvagg
Copy link
Member

rvagg commented Oct 20, 2019

node-test-commit-v8-linux
infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)

That's the benchmark machine, non-vm I think. I don't believe we have a strategy for dealing with that one because upgrades will change benchmark results. Bumping it to 18.04 would be nice. Similar story for test-nearform_intel-ubuntu1604-x64-{1,2} but we have the option of just adding in a new compiler I think. @nodejs/benchmarking, thoughts? Implications of doing a broad OS upgrade on these machines?

node-test-commit-linux-containered

This is odd because they're supposed to be all 18.04 as of last week, but it looks like I made a boo boo:

I'll get that sorted today.

node-test-commit-freebsd

🤷‍♂ anyone inclined to figure this out? I don't have time to work out how to upgrade this or get Ansible upgraded so my suggestion is to exclude it in https://github.com/nodejs/build/blob/master/jenkins/scripts/VersionSelectorScript.groovy for newer Node until it is sorted out

... the rest ...

The remaining ones are Ubuntu 16.04 and Debian 8. We have a decision point here:

  1. Introduce new machines that have newer compilers and maintain new and old, forking depending on the version of Node
  2. Get newer compilers onto these machines and use select-compiler.sh to do the work of switching depending on Node version (we've not done this for Debian & Ubuntu before). We can also do a similar job with build labels and scripting in Jenkins itself, but I select-compiler.sh seems to be a more contributor-friendly path.
  3. Remove Ubuntu 16.04 and Debian 8 from the supported list for Node 14 (or 13?) so we can drop them in VersionSelectorScript.groovy. This probably isn't a great idea since Ubuntu 16.04 is supported till 2021 and will go into extended (paid) support until 2024. Debian 8 still has until the middle of 2020, but maybe that's early enough to put it into retirement from our testing? Compiler upgrades on Debian are a nightmare.
  4. Upgrade compilers (where we can) and stop testing with older compilers for the older release lines. This would mean that, while we still say we support X compiler for Y release line, we just don't fully test that permutation in our infra.

I'm not in favour of adding more machines, that's more overhead for our stretched people-resources. Compiler switching is feasible but still a fair bit of work and adds a lot of complexity that adds to our maintenance burden (and the burden of onboarding new people, which constantly increases as we grow complexity).

@rvagg
Copy link
Member

rvagg commented Oct 21, 2019

gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) now on the sharedlibs / containered machines thanks to a proper Ubuntu 18.04 update.

@targos
Copy link
Member Author

targos commented Oct 21, 2019

node-test-commit-v8-linux
infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)

That's the benchmark machine, non-vm I think. I don't believe we have a strategy for dealing with that one because upgrades will change benchmark results. Bumping it to 18.04 would be nice. Similar story for test-nearform_intel-ubuntu1604-x64-{1,2} but we have the option of just adding in a new compiler I think. @nodejs/benchmarking, thoughts? Implications of doing a broad OS upgrade on these machines?

There's no ideal solution. For node-test-commit-v8-linux, we could probably just use other machines. I guess the benchmark one was chosen because it's powerful?
But we're going to have issues anyway if the benchmark machine is not updated. After V8 7.9 lands on master, we won't be able to run benchmarks on that branch.

@mhdawson
Copy link
Member

In terms of running on the benchmark machine I think we just did that to avoid affecting other builds. At this point I'd be in favor of building on other machines as well, but you are right that we'd still need to update the benchmark machine as well.

rvagg added a commit that referenced this issue Oct 23, 2019
@rvagg
Copy link
Member

rvagg commented Oct 23, 2019

@mhdawson I've removed infra-softlayer-ubuntu1404-x64-2 from node-test-commit-v8-linux and will apply #1975 to the nearform benchmarking machines. I've also moved the select-compiler.sh script run to just before make for all platforms (instead of being inside of the arch if's) so that when #1975 is merged it should select gcc-6 for master.

@rvagg
Copy link
Member

rvagg commented Oct 23, 2019

I think we have this sorted from our end:

node-test-commit-v8-linux

  • infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)

removed

  • test-nearform_intel-ubuntu1604-x64-1: (CXX=g++, 5.4.0)

updated and using gcc-6 for >= 12

node-test-commit-freebsd

  • test-digitalocean-freebsd11-x64-1: (CXX=ccache c++, 6.0.0)

#1973 but it doesn't sound like this is critical, clang 6 might be OK, but we can yank this when it breaks. I think we'll be yanking FreeBSD 11 soon anyway.

node-test-commit-custom-suites-freestyle

  • test-rackspace-ubuntu1604-x64-1: (CXX=ccache g++, 5.4.0)

Using gcc-6 for >=12

node-test-commit-arm

  • test-packetnet-ubuntu1604-arm64-2: (CXX=ccache g++, 5.4.0)

Using gcc-6 for >= 12

node-test-commit-linux-containered

  • test-joyent-ubuntu1804_sharedlibs_container-x64-4: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-10: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-1: (CXX=ccache g++, 5.4.0)
  • test-softlayer-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
  • test-digitalocean-ubuntu1804_sharedlibs_container-x64-7: (CXX=ccache g++, 5.4.0)

properly using Ubuntu 18.04 now and have a newer compiler

node-test-commit-linux

  • test-rackspace-debian8-x64-1: (CXX=ccache g++, 4.9.2)

Debian 8 excluded for Node > 12

  • test-rackspace-ubuntu1604-x64-2: (CXX=ccache g++, 5.4.0)

Using gcc-6 for >= 12

You have more green now in a new run of your V8 7.9 @ https://ci.nodejs.org/job/node-test-pull-request/26158/, the failures on Linux now are the two Alpine containers that aren't happy.

@rvagg rvagg closed this as completed Oct 23, 2019
@targos
Copy link
Member Author

targos commented Oct 23, 2019

Thanks a lot, Rod!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants