add python 3.9

[mattip]

Let's try this. manylinux has support, but maybe the test images do not (on linux). The chances Azure supports it for macOS and windows are slim.

[mattip]

Hmm. Lots of problems.

• There is no 3.9 on the Azure windows machines.
• There is a problem with the 32 bit builds (see below)
• There is a problem with the Azure linux builds: for some reason it is setting the python version in the host machine even though that is not needed. I have a fix for that.

32-bit linux builds are failing on assert_equal(myfunc(x_f64[::jj]), y_true64[::jj]) where myfunc is np.reciprocal and jj == -1. This is in test_avx_based_ufunc, and the machine is reporting NumPy CPU features: SSE SSE2 SSE3* SSSE3* SSE41* POPCNT* SSE42* AVX* F16C* FMA3* AVX2*, when the build machine reports

########### EXT COMPILER OPTIMIZATION ###########
Running from numpy source directory.
/opt/python/cp38-cp38/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution \
 option: 'define_macros'
  warnings.warn(msg)
CPU baseline  : 
  Requested   : 'min'
  Enabled     : SSE SSE2
  Flags       : -msse -msse2

CPU dispatch  : 
  Requested   : 'max -xop -fma4'
  Enabled     : SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
  Generated   : 
              : 
  SSE41       : SSE SSE2 SSE3 SSSE3
  Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1
  Detect      : SSE SSE2 SSE3 SSSE3 SSE41
              : numpy/core/src/umath/_umath_tests.dispatch.c
              : 
  AVX2        : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
  Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mavx2
  Detect      : AVX F16C AVX2
              : numpy/core/src/umath/_umath_tests.dispatch.c
CCompilerOpt._cache_write[784] : write cache to path -> /io/numpy/build/temp.linux-i686-3.8 \
/ccompiler_opt_cache_ext.py

########### CLIB COMPILER OPTIMIZATION ###########
CPU baseline  : 
  Requested   : 'max -xop -fma4'
  Enabled     : SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
  Generated   : none

@seiko2plus any ideas? I seem to have deja-vu, we had this problem and then fixed it once.

Also, on 32-bits, there are gcc warnings when building, I opened cython/cython#3768. I don't see this on the numpy/numpy builds, perhaps it is connected to using the manylinux1 docker.

[mattip]

It seems the Azure pipeline images do not have CPython3.9 beta versions. There is a github action that can install them, but it is only available on github actions, not Azure pipelines. Took me a few tries to understand the difference. xref actions/runner-images#912

[mattip]

I temporarily disabled Travis webhook while I play with Azure

[mattip]

Now the 3.9 test image (xenial from this repo) is missing libffi-dev, which is needed to build cffi, which is needed since there is no wheel for cffi. I wonder if we could split the test_requirements.txt into two parts: those that are critical for testing and those that are optional

[mattip]

this repo https://github.com/multi-build/docker-images

[mattip]

Weird, libffi-dev is supposed to be installed in that image.

[seiko2plus]

@mattip,

32-bit linux builds are failing on assert_equal(myfunc(x_f64[::jj]), y_true64[::jj]) where myfunc is np.reciprocal and jj == -1. This is in test_avx_based_ufunc

The compiler that had been used on this test doesn't support AVX2 throught target attribute __attribute__((target("avx2,fma"))).

The current avx2 tests assuming the compiler support of AVX2 on Linux, the best way to get rid of such errors like this is by moving the simd.inc.src to the new dispatcher as-is and then testing the compiler support through the new attrs __cpu_baseline__ and __cpu_dispatch__ before running accuracy tests.

Should I give it my first priority and work on it as a pre-step before totally moving to universal intrinsics?

[mattip]

Re-activated travis, I forgot I had disabled it.

[mattip]

The current avx2 tests assuming the compiler support of AVX2 on Linux

@seiko2plus what is the minimum gcc version for that support?

[seiko2plus]

@mattip, it seems attribute(target("...")) is kinda buggy on 32-bit mode, this issue will disappear starting from gcc6.
On the other side, 64-bit mode gcc5 and even 4.9(not all avx2 intrinsics) look normal.

[mattip]

Why are random people approving changes and what does that even mean?

[charris]

Why are random people approving changes

Beats me, I just ignore it.

[mattip]

Close/reopen to start over

[mattip]

On travis, the arm64 python 3.9 run is green (pypy3 is failing)
On azure, the linux64 and linux32 3.9 runs are green (manylinux1 32-bit linux is failing).
I am not sure how to build windows and macos, we need support from azure's images or a workaround.

[charris]

The failures aren't new. How about we make nightly wheels with this and tell people that want to test to use manylinux2010. I suspect that some of the many manylinux1 downloads are tests with pip pinned after the version 20.0 problems.

[mattip]

How about we make nightly wheels with this and tell people that want to test to use manylinux2010.

Ok, so this is ready for merging. Just they should only test with --only-binary numpy so they don't end up building from the sdist.

I suspect that some of the many manylinux1 downloads are tests with pip pinned after the version 20.0 problems

The same problem: if we don't have suitable binary wheels, their CI will suddenly start building from source.

Maybe we should send out a service announcement encouraging people to put numpy --only-binary=numpy in their requirements.txt

[charris]

Maybe we should send out a service announcement encouraging people to put numpy --only-binary=numpy in their requirements.txt

Sounds reasonable. Does pip check for supported versions before building from source? We don't yet list 3.9 in the CLASSIFIERS.

[charris]

Thanks Matti.