Compilation failure with gcc-6 ('_mm256_set1_pd': target specific option mismatch)

[ayounes-nviso]

Platform: Linux-Intel standard 16.04 (gcc-6)
OpenBLAS HEAD of develop: 6e2c494

Going back to this commit fixes the compilation error:
02ef20a Merge pull request #1786 from martin-frbg/immintrin

Full cmake build error:

In file included from /home/ayounes/devel/nv3dfi-build/linux_host/src/openblas_project-build/kernel/CMakeFiles/dgemm_kernel_SKYLAKEX.c:8:0:
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c: In function 'dgemm_kernel_SKYLAKEX':
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:48:7: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
ymm4 = _mm256_setzero_pd(); \

/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:1307:4: note: in expansion of macro 'INIT4x8'
  INIT4x8()
  ^~~~~~~
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:736:7: warning: AVX512F vector return without AVX512F enabled changes the ABI [-Wpsabi]
zmm4 = _mm512_setzero_pd();  \
~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:1565:4: note: in expansion of macro 'INIT8x1'
  INIT8x1()
  ^~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/immintrin.h:41:0,
               from /home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:36,
               from /home/ayounes/devel/nv3dfi-build/linux_host/src/openblas_project-build/kernel/CMakeFiles/dgemm_kernel_SKYLAKEX.c:8:
/usr/lib/gcc/x86_64-linux-gnu/6/include/avxintrin.h:1196:1: error: inlining failed in call to always_inline '_mm256_setzero_pd': target specific option mismatch
_mm256_setzero_pd (void)
^~~~~~~~~~~~~~~~~
In file included from /home/ayounes/devel/nv3dfi-build/linux_host/src/openblas_project-build/kernel/CMakeFiles/dgemm_kernel_SKYLAKEX.c:8:0:
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:55:8: note: called from here
ymm11 = _mm256_setzero_pd();  \
~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:1307:4: note: in expansion of macro 'INIT4x8'
  INIT4x8()
  ^~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/immintrin.h:41:0,
               from /home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_kernel_4x8_skylakex.c:36,
               from /home/ayounes/devel/nv3dfi-build/linux_host/src/openblas_project-build/kernel/CMakeFiles/dgemm_kernel_SKYLAKEX.c:8:
/usr/lib/gcc/x86_64-linux-gnu/6/include/avxintrin.h:1196:1: error: inlining failed in call to always_inline '_mm256_setzero_pd': target specific option mismatch
_mm256_setzero_pd (void)
^~~~~~~~~~~~~~~~~

[brada4]

For a quick fix you can try to disable AVX512 extension usage by adding NO_AVX512=1 to command line or Makefile.rule.
Could you provide some system information - 16.04 is Ubuntu I assume? Intel - is it x86_64 ?
Default gcc should be gcc v5 - does it work?

[martin-frbg]

target specific option mismatch error suggests that gcc is missing an option like -march=skylake-avx512 here. The Makefile build was changed fairly recently to add it automatically, but it could be that the cmake build was missed.

[brada4]

Indeed make build proceeds just fine (like ubuntu 16/04 installed hour ago).
Default gcc-5 does not support AVX512 and that gets detected poperly with fallback to haswell
PPA gcc-6 gcc-7 gcc-8 work just fine

[ayounes-nviso]

Could you provide some system information - 16.04 is Ubuntu I assume? Intel - is it x86_64 ?

yes. Default gcc should be gcc v5 - does it work?

Défaut is 6.4 in out Docker... And yes, or works fine.

[martin-frbg]

cmake/system_check.cmake just needs an else() set (CCOMMON_OPT "${CCOMMON_OPT} -march=skylakex-avx512 added to the if clause near its end. I will create a PR soon, in the meantime I think you should get it working by just adding this -march= option to your CFLAGS

[brada4]

@martin-frbg yes, that is missing. after adding march flag allows cmake build to complete on gcc-6 and above.

[ayounes-nviso]

Thanks a lot for the prompt fix!

[ayounes-nviso]

I am sorry but this should be open again:
[1469/6999] Building C object kernel/CMake...X.dir/CMakeFiles/dgemm_otcopy_SKYLAKEX.c.o
FAILED: kernel/CMakeFiles/kernel_SKYLAKEX.dir/CMakeFiles/dgemm_otcopy_SKYLAKEX.c.o
/usr/bin/cc -I/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src -Ikernel_config/SKYLAKEX -fPIC -DDYNAMIC_ARCH -DNO_LAPACK -DNO_LAPACKE -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -O3 -DNDEBUG -fPIC -DBUILD_KERNEL -DTABLE_NAME=gotoblas_SKYLAKEX -DTS=_SKYLAKEX -MD -MT kernel/CMakeFiles/kernel_SKYLAKEX.dir/CMakeFiles/dgemm_otcopy_SKYLAKEX.c.o -MF kernel/CMakeFiles/kernel_SKYLAKEX.dir/CMakeFiles/dgemm_otcopy_SKYLAKEX.c.o.d -o kernel/CMakeFiles/kernel_SKYLAKEX.dir/CMakeFiles/dgemm_otcopy_SKYLAKEX.c.o -c kernel/CMakeFiles/dgemm_otcopy_SKYLAKEX.c
In file included from kernel/CMakeFiles/dgemm_otcopy_SKYLAKEX.c:9:0:
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_tcopy_8_skylakex.c: In function ‘dgemm_otcopy_SKYLAKEX’:
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_tcopy_8_skylakex.c:87:9: warning: AVX512F vector return without AVX512F enabled changes the ABI [-Wpsabi]
row1 = _mm512_loadu_pd(aoffset1);
~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_tcopy_8_skylakex.c:119:7: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
row1 = _mm256_loadu_pd(aoffset1);

In file included from /usr/lib/gcc/x86_64-linux-gnu/7/include/immintrin.h:45:0,
               from /home/ayounes/devel/nv3dfi/ext/pkg-openblas/src/kernel/x86_64/dgemm_tcopy_8_skylakex.c:41,
               from kernel/CMakeFiles/dgemm_otcopy_SKYLAKEX.c:9:
/usr/lib/gcc/x86_64-linux-gnu/7/include/avx512fintrin.h:5722:1: error: inlining failed in call to always_inline ‘_mm512_storeu_pd’: target specific option mismatch
_mm512_storeu_pd (void *__P, __m512d __A)
^~~~~~~~~~~~~~~~

[martin-frbg]

Sorry, closed automatically as I had written "should fix" in the commit message. Not sure what went wrong but probably some silly mistake again.

[martin-frbg]

Seems modifications to CCOMMON_OPT are ignored at that point in the build sequence, setting KERNEL_DEFINITIONS in #1812 instead.

[ayounes-nviso]

#1812 fixes the compilation for me.

[martin-frbg]

Thanks. I had some misgivings that it might not work with non-DYNAMIC, TARGET=SKYLAKEX builds but apparently there was something else wrong with my test. (Not surprising as I was doing this on the side at work)

[ayounes-nviso]

I can also confirm that the crash is gone.
But as I am using DYNAMIC and I don't have any Skylakex I cannot test further than that.
Thanks anyway.

[martin-frbg]

I do not have AVX512 hardware at the moment either, but at least I can see that a TARGET=SKYLAKEX build gets the -march=skylake-avx512 option now as it should. Thanks for confirming that the original issue is resolved.