Testing if python 3.8 is available. #62

charris · 2019-10-03T17:51:55Z

No description provided.

mattip · 2019-10-03T22:18:39Z

.travis.yml

@@ -46,6 +46,9 @@ matrix:
      env:
        - MB_PYTHON_VERSION=3.7
        - PLAT=i686
+    - os: linux
+      env:
+        - MB_PYTHON_VERSION=3.8


Could you try here to use the manylinux2010 image by adding MB_ML_VER=2010? In order to use this you will also need to update the multibuild subrepo to the HEAD of the devel branch

Sadly, it failed at the same spot, it cannot find pip.

Did download 2010 however.

The path in the docker image is /opt/python/cp38-cp38, multibuild expects a m suffix /opt/python/cp38-cp38m

I could test locally on ubuntu 18.04 with

##!bash set -x REPO_DIR=numpy # Also see DAILY_COMMIT below BUILD_COMMIT=master BUILD_DEPENDS=cython TEST_DEPENDS=pytest PLAT=x86_64 UNICODE_WIDTH=32 WHEELHOUSE_UPLOADER_USERNAME=travis-worker MB_PYTHON_VERSION=3.8 MB_ML_VER=2010 source multibuild/common_utils.sh source multibuild/travis_steps.sh before_install clean_code $REPO_DIR $BUILD_COMMIT ./patch_code.sh $REPO_DIR if [ "$TRAVIS_OS_NAME" == "linux" ]; then export CFLAGS=${CFLAGS}" -Wno-sign-compare -Wno-unused-result\ -Wno-strict-aliasing"; fi build_wheel $REPO_DIR $PLAT

and to debug the docker I added set -x to the top of multibuild/docker_build_wrap.sh

What is the difference between a python with and without the m suffix?

charris · 2019-10-04T12:12:36Z

What is the difference between a python with and without the m suffix?

Don't know, but I recall seeing somewhere that the "m" was going away.

mattip · 2019-10-04T12:30:52Z

the m was removed fromget_abi_tag in wheel.pep425tags.py by pypa/wheel#303 for python3.8 and up.

mattip · 2019-10-04T12:48:19Z

If I locally modify multibuild as per matthew-brett/multibuild#262, I get further, but auditwheel fails, it needs libgfortran.so.3:

+ cp dist/numpy-1.18.0.dev0+ec24179-cp38-cp38-linux_x86_64.whl /io/wheelhouse
+ repair_wheelhouse /io/wheelhouse
+ local in_dir=/io/wheelhouse
+ local out_dir=/io/wheelhouse
+ for whl in '$in_dir/*.whl'
+ [[ /io/wheelhouse/numpy-1.18.0.dev0+ec24179-cp38-cp38-linux_x86_64.whl == *none-any.whl ]]
+ auditwheel repair /io/wheelhouse/numpy-1.18.0.dev0+ec24179-cp38-cp38-linux_x86_64.whl -w /io/wheelhouse/
INFO:auditwheel.main_repair:Repairing numpy-1.18.0.dev0+ec24179-cp38-cp38-linux_x86_64.whl
Traceback (most recent call last):
  File "/usr/local/bin/auditwheel", line 10, in <module>
    sys.exit(main())
  File "/opt/_internal/cpython-3.6.9/lib/python3.6/site-packages/auditwheel/main.py", line 50, in main
    rval = args.func(args, p)
  File "/opt/_internal/cpython-3.6.9/lib/python3.6/site-packages/auditwheel/main_repair.py", line 83, in execute
    update_tags=args.UPDATE_TAGS)
  File "/opt/_internal/cpython-3.6.9/lib/python3.6/site-packages/auditwheel/repair.py", line 86, in repair_wheel
    soname)
ValueError: Cannot repair wheel, because required library "libgfortran.so.3" could not be located

mattip · 2019-10-06T05:26:20Z

can you update the multibuild subrepo to f0bff09 which will pull in a change to the way it processes the m abi suffix?

charris · 2019-10-06T13:26:57Z

Goes further, but shows the same error you found locally.

ValueError: Cannot repair wheel, because required library "libgfortran.so.3" could not be located

I wonder if that is a manylinux2010 problem?

charris · 2019-10-06T13:28:35Z

It isn't like NumPy needs gfortran. Maybe BLAS? We can disable the OpenBLAS library and see if that fixes things.

mattip · 2019-10-06T17:02:58Z

OpenBLAS needs gfortran. The multilinux2010 docker image has /usr/lib64/libgfortran.so.5, not what auditwheel is looking for.

matthew-brett · 2019-10-06T17:47:22Z

I guess we need to build OpenBLAS on manylinux2010 then? Or can we just link the libgfortran.so.5 binary to the right name? Does it contain all the symbols / versions we use when building from a Manylinux1 container?

charris · 2019-10-06T18:19:42Z

Little googling seems to imply that we either need to compile with later fortran or install libgfortran.so.3. Not sure if the wheel will end up containing two fortran libraries if we do that. The best bet would seem to be making OpenBLAS libraries that are compatible with the fortran in manylinux2010 if we move to that.

mattip · 2019-10-06T18:21:00Z

we could ask the manylinux2010 people to provide libgfortran.so.3

mattip · 2019-10-06T18:21:50Z

it is in the libgfortran-4 rpm

charris · 2019-10-06T18:34:41Z

ISTR that backward compatibility was broken in some gfortran release, although it was claimed at some point that it would be maintained. The safe bet is to use the library that goes with the compiler.

mattip · 2019-10-06T18:35:51Z

Adding a pre_build function to create a symbolic link for libfortran.so.3 got me passed the auditwheel stage, but it is an awful hack since it rewrites the libblas to use libfortran.so.5. I don't know if that works.

diff --git a/config.sh b/config.sh
index 0329262..7659d40 100644
--- a/config.sh
+++ b/config.sh
@@ -48,3 +48,12 @@ function run_tests {
     # Show BLAS / LAPACK used
     python -c 'import numpy; numpy.show_config()'
 }
+
+function pre_build {
+    if [ -z "$IS_OSX" ]; then
+        # manylinux2010 image uses gfortran.so.5, we still need gfortran.so.3
+        if [ "$MB_ML_VER" == "2010" ]; then
+            ln -s /usr/lib64/libgfortran.so.5 /usr/lib64/libgfortran.so.3
+        fi
+    fi
+}

charris · 2019-10-06T18:37:42Z

My machine has

compat-libgfortran-41.i686 : Compatibility Fortran 95 runtime library version 4.1.2
compat-libgfortran-41.x86_64 : Compatibility Fortran 95 runtime library version 4.1.2

since it is now at version 5.

mattip · 2019-10-06T18:40:38Z

Hopefully we can separate between manylinux1 -> manylinux2010 transition (which we should do across the board and not just for 3.8) and this PR. pypa/manylinux#360 to get 3.8 on manylinux1 might be merged soon.

mattip · 2019-10-06T18:41:17Z

compat-libgfortran-41.x86_64

is that CentOS ?

charris · 2019-10-06T18:44:16Z

Fedora 30. You might try yum search libgfortran if you are in CentOS (Old Redhat derived).

mattip · 2019-10-07T09:24:29Z

pypa/manylinux#360 was merged, can you try removing the MB_ML_VER=2010 spec from the .travis.yml file?

charris · 2019-10-07T12:28:13Z

Seems like the default python3.3 in the docker image is being used.

charris · 2019-10-07T12:30:18Z

Does mathew-brett/trusty need an update?

charris · 2019-10-07T12:57:19Z

It builds, the wheel is created, but fails at install_run $PLAT, which seems to be a problem with matthew-brett/trusty.

mattip · 2019-10-07T15:57:10Z

xref matthew-brett/trusty#5 for 64-bit builds

charris · 2019-10-07T20:14:15Z

Close/reopen.

mattip · 2019-10-07T21:54:07Z

The 64-bit image was updated, the 32-bit still not

Testing if python 3.8 is available.

96b450f

mattip mentioned this pull request Oct 3, 2019

Support Python 3.8 builds multi-build/multibuild#261

Closed

mattip reviewed Oct 3, 2019

View reviewed changes

charris added 2 commits October 3, 2019 18:30

Update multibuild to latest devel.

4a7976f

test manylinux2010

c4f5c42

charris added 4 commits October 6, 2019 07:10

Update multibuild to f0bff09.

1b365bb

Disable all but 3.8 builds.

0d9f0e7

Fix .appveyor parse error, maybe.

0b995f0

Try another appveyor fix.

29c2882

Try latest manylinux1.

374289d

Be explicit about manylinux version=1

53c211d

charris closed this Oct 7, 2019

charris reopened this Oct 7, 2019

Try i386

f63ac91

try again

e61d303

charris closed this Oct 7, 2019

charris deleted the test-python-3.8 branch October 7, 2019 23:41

This was referenced Oct 9, 2019

Creating 3.8.0rc1 wheels MacPython/scipy-wheels#49

Closed

Wheels for 3.8.0rc1 MacPython/pandas-wheels#61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing if python 3.8 is available. #62

Testing if python 3.8 is available. #62

charris commented Oct 3, 2019

mattip Oct 3, 2019

charris Oct 4, 2019

charris Oct 4, 2019

mattip Oct 4, 2019

mattip Oct 4, 2019

mattip Oct 4, 2019

charris commented Oct 4, 2019

mattip commented Oct 4, 2019

mattip commented Oct 4, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

matthew-brett commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 7, 2019

charris commented Oct 7, 2019

charris commented Oct 7, 2019

charris commented Oct 7, 2019

mattip commented Oct 7, 2019

charris commented Oct 7, 2019

mattip commented Oct 7, 2019

Testing if python 3.8 is available. #62

Testing if python 3.8 is available. #62

Conversation

charris commented Oct 3, 2019

mattip Oct 3, 2019

Choose a reason for hiding this comment

charris Oct 4, 2019

Choose a reason for hiding this comment

charris Oct 4, 2019

Choose a reason for hiding this comment

mattip Oct 4, 2019

Choose a reason for hiding this comment

mattip Oct 4, 2019

Choose a reason for hiding this comment

mattip Oct 4, 2019

Choose a reason for hiding this comment

charris commented Oct 4, 2019

mattip commented Oct 4, 2019

mattip commented Oct 4, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

matthew-brett commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 6, 2019

mattip commented Oct 6, 2019

charris commented Oct 6, 2019

mattip commented Oct 7, 2019

charris commented Oct 7, 2019

charris commented Oct 7, 2019

charris commented Oct 7, 2019

mattip commented Oct 7, 2019

charris commented Oct 7, 2019

mattip commented Oct 7, 2019