Reorganize Jenkins test stages

[rok-cesnovar]

Summary

Reorganizes Jenkins stages as suggested here: https://discourse.mc-stan.org/t/bringing-math-testing-time-resources-down/12885/5

The most compute intensive stages are organized as follows:

• Headers check
• Linux Unit with MPI
• Parallel stage with: Full unit with GPU (with OpenCL header check), Distribution tests, Threading tests, Windows Headers & Unit, Windows Threading

Reasons:

• Most of the time Linux Unit with MPI finishes in 20 minutes (unless if it gets assigned to one specific worker, then its an hour) and if I am not mistaken the largest number of the executors are Linux so this should not be such a bottleneck
• a lot of jobs wait a lot of time for the GPU device. This way we reduce the executions of GPU jobs (jobs that will fail on linux will not run on GPU)
• the wait for GPU executors is done in parallel with the distribution tests
• if Linux unit tests pass most of the time the tests will go green (provided there arent any false positives due to CI issues)

Tests

Describe the new tests with the pull request.

For bug fixes there should be a new test that would fail if the patch weren't in place (so that the bug could be caught if it comes up again).

For new features there should be at least one test showing the expected behavior and one test that demonstrates error handling. Be aware the reviewer will very likely ask for more tests than this, but it's a start.

Side Effects

/

Checklist

• Copyright holder: Rok Češnovar

  The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
  - Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
  - Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

• the basic tests are passing

  • unit tests pass (to run, use: ./runTests.py test/unit)
  • header checks pass, (make test-headers)
  • dependencies checks pass, (make test-math-dependencies)
  • docs build, (make doxygen)
  • code passes the built in C++ standards checks (make cpplint)

• the code is written in idiomatic C++ and changes are documented in the doxygen

• the new changes are tested

[rok-cesnovar]

I dropped the ball on this one, but better late then never. Tagging a few people that were interested in this 2 months ago @martinmodrak @serban-nicusor-toptal @syclik @SteveBronder

[stan-buildbot]

---

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
gp_pois_regr/gp_pois_regr.stan	4.83	4.86	0.99	-0.59% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.02	0.02	1.01	1.13% faster
eight_schools/eight_schools.stan	0.09	0.09	1.0	0.18% faster
gp_regr/gp_regr.stan	0.22	0.22	1.01	0.88% faster
irt_2pl/irt_2pl.stan	6.45	6.45	1.0	-0.07% slower
performance.compilation	87.91	86.7	1.01	1.38% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	7.55	7.52	1.0	0.47% faster
pkpd/one_comp_mm_elim_abs.stan	21.34	21.4	1.0	-0.29% slower
sir/sir.stan	93.22	93.74	0.99	-0.56% slower
gp_regr/gen_gp_data.stan	0.05	0.05	1.03	3.34% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.96	2.96	1.0	-0.01% slower
pkpd/sim_one_comp_mm_elim_abs.stan	0.31	0.31	1.0	0.19% faster
arK/arK.stan	1.74	1.74	1.0	0.11% faster
arma/arma.stan	0.67	0.66	1.0	0.3% faster
garch/garch.stan	0.51	0.52	1.0	-0.49% slower
Mean result: 1.00408920995

---

Jenkins Console Log
Blue Ocean
Commit hash: 773afed

---

Machine information

ProductName: Mac OS X ProductVersion: 10.11.6 BuildVersion: 15G22010

CPU:
Intel(R) Xeon(R) CPU E5-1680 v2 @ 3.00GHz

G++:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

Clang:
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

[stan-buildbot]

---

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
gp_pois_regr/gp_pois_regr.stan	4.88	4.88	1.0	-0.01% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.02	0.02	0.99	-0.66% slower
eight_schools/eight_schools.stan	0.09	0.09	0.98	-1.6% slower
gp_regr/gp_regr.stan	0.22	0.22	1.0	-0.29% slower
irt_2pl/irt_2pl.stan	6.46	6.45	1.0	0.17% faster
performance.compilation	89.04	86.6	1.03	2.75% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	7.56	7.54	1.0	0.29% faster
pkpd/one_comp_mm_elim_abs.stan	20.04	21.29	0.94	-6.21% slower
sir/sir.stan	97.51	94.05	1.04	3.55% faster
gp_regr/gen_gp_data.stan	0.05	0.05	1.0	0.26% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.95	2.96	1.0	-0.24% slower
pkpd/sim_one_comp_mm_elim_abs.stan	0.31	0.33	0.92	-8.81% slower
arK/arK.stan	1.76	1.75	1.01	0.84% faster
arma/arma.stan	0.67	0.66	1.02	1.73% faster
garch/garch.stan	0.52	0.51	1.02	1.96% faster
Mean result: 0.996743017416

---

Jenkins Console Log
Blue Ocean
Commit hash: 773afed

---

Machine information

ProductName: Mac OS X ProductVersion: 10.11.6 BuildVersion: 15G22010

CPU:
Intel(R) Xeon(R) CPU E5-1680 v2 @ 3.00GHz

G++:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

Clang:
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

[SteveBronder]

I'm cool with this! The Jenkinsfile looks fine to me, but if @serban-nicusor-catena has any time to take a look at double check it that would be great

[serban-nicusor-toptal]

Looks good to me, many thanks for looking into it @rok-cesnovar !