Add centos7 PPCle machines

[mhdawson]

If possible we'd like to build on Centos76 for PPC now that centos is available at OSU. Starting with using them for 12.x and above and then seeing if we can use them for older releases as well.

• Add 2 test centos machines (test-osuosl-centos7-ppc64-le-1, test-osuosl-centos7-ppc64-le-2)
• Configure test machines with existing Ansible scripts if possible
• Validate that compiler is installed properly: devtoolset-6 installed
• Update ansible scripts if necessary
• Update regression tests so that tests also run on centos76 for 12.x+, validate all passes
• Add 1 release centos machine (release-osuosl-centos7-ppc64-le-1)
• update select-compiler.sh: select devtoolset-6 ONLY for centos7-ppc64le (if it's ubuntu, continue ot use gcc-6)
• Configure release machine with Ansible
• Add release machine to release CI.
• ... add additional required keys to release machine etc.
• Do test builds with cloned release job where we build releases on centos7 for 13+ instead of unbuntu
• Land PR for groovy selector to start doing releases on centos7 for 13+ instead of ubuntu
• Audit community test jobs to make sure we have added centos7 to all test jobs
• Identify next release, and plan switch over so that we get X days of nightly build/testing before next
  real release.
• Land PR for groovy selector to start doing releases on centos7 for 12+ instead of ubuntu

[refack]

Is the plan to (if all goes well) use these and retire the ubuntu14-ppc64-le workers?

[mhdawson]

@refack assuming we convince ourselves that older binaries still run on all target OS's (of which I'm hopeful) then yes but I don't want to rush that so my plan would be to do that slowly.

[sam-github]

@mhdawson Tried to tick off some of the items above, but only build-wg members can do that.

What are the "regression tests"?

[sam-github]

@refack You probably already know this, but for anyone else: ubuntu 14.04 is aging out of LTS soon, so yes, we want to move to centos7 because it's support horizon will last long enough for 12.x.

To replace the ubuntu14.04 with centos7 for < 12.x, I need to find gcc 4.8, and 4.9 (at least) for them, and I haven't had any luck with that yet. Worst case is we'll have to build them ourselves via ansible, which would be frustrating.

[mhdawson]

And to set expectations if we can/will retire ubuntu14.04 it will be after 12.x not before....

[sam-github]

@mhdawson #1763 is ready for review. I am a bit fuzzy on the rollover plan, but I believe it reflects step-1, in my comment above.

I think the next step is to merge it (you have to do), then configure jenkins (you have to do).

Once ci is running on centos7, I can check the builds.

Once that is happening, is the next step that the groovy exclude script is changed so we stop building ubuntu-ppc64le for node >- 12.x? I guess that will be a new PR, I can do it.

[mhdawson]

I think the PR is ready to go but I think we should wait until after 12.x goes out next week to land it. Will plan to look at that on Wednesday.

[richardlau]

@mhdawson @sam-github @AshCripps I think this has all been done now and can be closed?

[mhdawson]

@sam-github can you confirm and close.