Build failure on circle-ci: f951: Error: bad value (skylake-avx512) for -march= switch

[iMichka]

Hi

Release 0.3.4 fails to build on Linux (on our CI), with gcc5 and gcc6. (related issue, https://github.com/Linuxbrew/homebrew-core/pull/10455).

CPU: 36-core 64-bit skylake
Kernel: Linux 4.4.0-139-generic x86_64 GNU/Linux
OS: Ubuntu 16.04.5 LTS (xenial)
Host glibc: 2.23

Last 150 lines from /home/linuxbrew/.cache/Homebrew/Logs/openblas/01.make:
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o spotrf2.o spotrf2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgetrf2.o sgetrf2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbbrd.o sgbbrd.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbcon.o sgbcon.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbequ.o sgbequ.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbrfs.o sgbrfs.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbsv.o sgbsv.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbsvx.o sgbsvx.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbtf2.o sgbtf2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbtrf.o sgbtrf.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgbtrs.o sgbtrs.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgebak.o sgebak.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgebal.o sgebal.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgebd2.o sgebd2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgebrd.o sgebrd.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgecon.o sgecon.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeequ.o sgeequ.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgees.o sgees.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeesx.o sgeesx.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeev.o sgeev.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeevx.o sgeevx.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgehd2.o sgehd2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgehrd.o sgehrd.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgelq2.o sgelq2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgelqf.o sgelqf.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgels.o sgels.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgelsd.o sgelsd.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgelss.o sgelss.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgelsy.o sgelsy.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeql2.o sgeql2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeqlf.o sgeqlf.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeqp3.o sgeqp3.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeqr2.o sgeqr2.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeqr2p.o sgeqr2p.f
gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeqrf.o sgeqrf.f
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'sbdsvdx.o' failed
make[2]: *** [sbdsvdx.o] Error 1
make[2]: *** Waiting for unfinished jobs....
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'sgbcon.o' failed
make[2]: *** [sgbcon.o] Error 1
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'spotrf2.o' failed
make[2]: *** [spotrf2.o] Error 1
f951: Error: bad value (skylake-avx512) for -march= switch
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'sgbequ.o' failed
make[2]: *** [sgbequ.o] Error 1
Makefile:602: recipe for target 'sgbsv.o' failed
make[2]: *** [sgbsv.o] Error 1
f951: Error: bad value (skylake-avx512) for -march= switch
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'sgeequ.o' failed
make[2]: *** [sgeequ.o] Error 1
Makefile:602: recipe for target 'sgeevx.o' failed
make[2]: *** [sgeevx.o] Error 1
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'sgbrfs.o' failed
make[2]: *** [sgbrfs.o] Error 1
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'sgbbrd.o' failed
make[2]: *** [sgbbrd.o] Error 1
f951: Error: bad value (skylake-avx512) for -march= switch

Previous openblas versions were fine.

[sjackman]

This issue occurs when compiling OpenBLAS on an AVX-512 capable machine like Skylake with a version of GCC that does not understand march=skylake-avx512, like GCC 5, which ships with Ubuntu 16.04 LTS (Xenial Xerus). CircleCI recently upgraded their hardware to AVX-512 machines.

[martin-frbg]

Seems to me your gcc and gfortran are different versions - the gcc recent enough to accept the flag (so the compile test in c_check passes and NO_AVX512 does not get set), but gfortran too old (no separate avx512 test in f_check here as mixing different generations of the two compilers is a bad idea anyway).

[sjackman]

In that particular failure that's the issue. Thanks for that pointer. We missed that.

Here's a build failure where both gcc and gfortran are the same version, and it fails.

gfortran -O2 -Wall -m64 -fopenmp -fPIC -march=skylake-avx512 -c -o sgeqrf.o sgeqrf.f
f951: Error: bad value (skylake-avx512) for -march= switch
Makefile:602: recipe for target 'sbdsvdx.o' failed

https://circleci.com/gh/Linuxbrew/homebrew-core/22607

[martin-frbg]

This certainly should not happen. As I wrote above, there is a compile test that gets run very early in the build process to see if AVX512 support is available. If the test fails, NO_AVX512 is set to 1 and SkylakeX is subsequently treated like Haswell.
I seem to remember the original test case did not probe all potential problems, and you seem to be
still building 0.3.3 rather than the recent 0.3.4 release. Would it be possible for you to update to the
current version ?
(Unfortunately while the ci log you linked to is long, it does not contain the full information from the build - though with the high core count it is probably possible that both gcc and gfortran failed in parallel and only the gfortran errors appear in the abbreviated log. Still I wonder what c_check and
subsequent invocations of gcc "saw" )

[sjackman]

This certainly should not happen. As I wrote above, there is a compile test that gets run very early in the build process to see if AVX512 support is available. If the test fails, NO_AVX512 is set to 1 and SkylakeX is subsequently treated like Haswell.

Can we set NO_AVX512 manually? Is either ./configure NO_AVX512=1 or make NO_AVX512=1 expected to work?

Would it be possible for you to update to the current version ?

Yes. This build is in progress.
https://circleci.com/gh/Linuxbrew/homebrew-core/23003

Unfortunately while the ci log you linked to is long, it does not contain the full information from the build

What information do you need? I can reproduce this build failure locally outside of CircleCI on an AVX-512 machine at work.

[sjackman]

On a Haswell machine at work I'm seeing a different build failure.

gcc-5 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=2 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.4\" -march=skylake-avx512 -DASMNAME=sgemm_itcopy_SKYLAKEX -DASMFNAME=sgemm_itcopy_SKYLAKEX_ -DNAME=sgemm_itcopy_SKYLAKEX_ -DCNAME=sgemm_itcopy_SKYLAKEX -DCHAR_NAME=\"sgemm_itcopy_SKYLAKEX_\" -DCHAR_CNAME=\"sgemm_itcopy_SKYLAKEX\" -DNO_AFFINITY -DTS=_SKYLAKEX -I.. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_SKYLAKEX -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/x86_64/sgemm_tcopy_16_skylakex.c -o sgemm_itcopy_SKYLAKEX.o
In file included from /home/linuxbrew/.linuxbrew/Cellar/gcc/5.5.0_4/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/include/immintrin.h:45:0,
                 from ../kernel/x86_64/sgemm_kernel_16x4_skylakex.c:57:
../kernel/x86_64/sgemm_kernel_16x4_skylakex.c: In function 'sgemm_kernel_SKYLAKEX':
/home/linuxbrew/.linuxbrew/Cellar/gcc/5.5.0_4/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/include/avx512fintrin.h:239:1: error: inlining failed in call to always_inline '_mm512_setzero_ps': target specific option mismatch
 _mm512_setzero_ps (void)
 ^
../kernel/x86_64/sgemm_kernel_16x4_skylakex.c:90:8: error: called from here
  row3d = _mm512_setzero_ps();     \
        ^
../kernel/x86_64/sgemm_kernel_16x4_skylakex.c:797:4: note: in expansion of macro 'INIT64x4'
    INIT64x4()
    ^

https://gist.github.com/sjackman/aaf78d42c2bff38bc16aeca085e9996e#file-01-make-L1600-L1612

[martin-frbg]

Yes I think make NO_AVX512=1 would work.
Build failure on Haswell could be related to gcc bug 78451 i.e. you may need to have gcc 7.x for SkylakeX builds now.

[sjackman]

We'll try make NO_AVX512=1 and see if it helps.

Bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78451 indicates that this issue was resolved in GCC 7.x. It doesn't indicate if there's a work-around for earlier versions of GCC. Previous versions of OpenBLAS compiled fine on Haswell with GCC 5. Do you know if there's a work around for compiling OpenBLAS on Haswell with GCC 5?

[brada4]

You need matching GCC an GFORTRAN, v5 would disable AVX-512 code reducing it to HASWELL code, v6 and later (matching versions) from Ubuntu compiler PPA wil compile everything
make CC=gcc-8 FC=gfortran-8
#1797
@martin-frbg do you think AVX-512 compiler support fixups should be described in FAQ?

[sjackman]

This build on Haswell uses gcc and gfortran 5.5.0 but fails. /usr/bin/gcc is 5.4.0 and is not being used. See https://gist.github.com/sjackman/aaf78d42c2bff38bc16aeca085e9996e#file-01-make-L1600-L1612

[brada4]

There is no mention of make parameters in whole log.
There is at least USE_OPENMP=1 passed to make
5.5 < 6.0 ? or you see otherwise?

[sjackman]

There is no mention of make parameters in whole log.

See https://gist.github.com/sjackman/aaf78d42c2bff38bc16aeca085e9996e#file-01-make-L3-L8

make CC=gcc-5 FC=gfortran libs netlib shared

gcc-5 is GCC 5.5.0 and gfortran is also 5.5.0.

There is at least USE_OPENMP=1 passed to make

I don't believe so.

5.5 < 6.0 ? or you see otherwise?

We're using gcc 5.5.0 and gfortran 5.5.0.

@iMichka had one CircleCI build that experimented with using GCC 6.0. That's not the build shown in this gist.

[brada4]

I dont know what your compiler does wrong.
gcc 5.3.1 and gfortran 5.3.1 included with your Linux distribution correctly disables AVX-512.

[martin-frbg]

What fails on Haswell is the SkylakeX part of the DYNAMIC_ARCH support, which is the only component
that currently uses AVX512 instructions. Adding NO_AVX512=1 to the make options should work around this. (If it does not, the alternative would be to supply a DYNAMIC_LIST argument with a whitespace-separated list of DYNAMIC_ARCH targets to include instead of the hardcoded one. This was added some months ago to solve issue #1595)

[sjackman]

There is at least USE_OPENMP=1 passed to make

I don't believe so.

The environment variables DYNAMIC_ARCH=1 and USE_OPENMP=1 are defined.
See https://github.com/Linuxbrew/homebrew-core/blob/b6a0f6b3255441d34b7a7ab76cc5d36961cd6414/Formula/openblas.rb#L29-L34