Parallelized Build / CI

[WillAyd]

Modeled after cython/cython#3187 which in 0.29.14 added parallel support

A very slight boost to timing from a clean directory (bottleneck is in the Cythonize calls in both cases)

# Master
$ time python setup.py build_ext --inplace -j 4 : 
62.93s user 3.18s system 97% cpu 1:07.77 total

# this branch
$ time python setup.py build_ext --inplace -j 4:
58.87s user 3.32s system 105% cpu 59.030 total

[jbrockmendel]

LGTM

[WillAyd]

I think could also reduce the cythonize bottleneck if using the nthreads argument:

https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#Cython.Build.cythonize

Locally that yielded this for a clean build:

69.07s user 3.75s system 334% cpu 21.783 total

So 21 seconds from scratch isn't bad; might help a lot to reduce time to run ASVs

[jbrockmendel]

that would be a nice speedup. im not clear on what the difference is between nthreads vs "-j"

[jbrockmendel]

ok, it looks like passing nthreads to cythonize affects the cython->C compilation, and the "-j 4" affects the C->so compilation

[WillAyd]

Open to suggestions on where to place this, but reusing the argument for parallel compliation in the calls to Cythonize helps a lot. Updated benchmarks:

$ time python setup.py build_ext --inplace --parallel=4
70.69s user 3.88s system 311% cpu 23.951 total
$ time python setup.py build_ext --inplace -j4
69.88s user 3.80s system 322% cpu 22.817 total

[WillAyd]

I'd roughly say this saves about 4-5 minutes on CI where it works. Have to see what's going on with Linux 36

Also not sure if Windows will really support this, at least not without some upstream changes. Earlier attempts throw a warning about multiprocessing freeze_support; put on the back burner for now

[jbrockmendel]

the CI errors all look unrelated, just started re-run

[jreback]

lgtm thanks @WillAyd

[WillAyd]

I think this might be responsible for some 36 errors on the 32 bit build with environment setup this morning. I'll revert at least the CI piece so we can get that green