xmeshfem3D run error

[ibearye]

When I run the example in $SPECFEM3D/EXAMPLES/meshfem3D_examples/regular_element_mesh, some errors was thrown. I run the example by run run_this_example.sh shell.

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:

Could not print backtrace: executable file is not an executable
#0  0x1033a1147
#1  0x1033a01db
#2  0x1ad6694c3
#3  0x20f26eef3
#4  0x20f271e83
#5  0x1b5f487bb
#6  0x1b5f485f3
#7  0x1ad48e1c7
#8  0x1ad49d413
#9  0x1b601445b
#10  0x1b5f48413
#11  0x1b5f70e33
#12  0x1ad48e1c7
#13  0x1ad48fa2f
#14  0x1b5f481e3
#15  0x1049b4437
#16  0x1049ba5c7
#17  0x20eb1d14b
./run_this_example.sh: line 51: 37423 Illegal instruction: 4  ./bin/xmeshfem3D

environment:
MacOS 12.3 , M1 Silicon.

open-mpi: stable 4.1.3 (bottled), HEAD
High performance message passing library
https://www.open-mpi.org/
Conflicts with:
  mpich (because both install MPI compiler wrappers)
/opt/homebrew/Cellar/open-mpi/4.1.3 (758 files, 17.6MB) *
  Poured from bottle on 2022-04-06 at 12:57:14
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/open-mpi.rb
License: BSD-3-Clause
==> Dependencies
Required: gcc ✔, hwloc ✔, libevent ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 22,562 (30 days), 62,015 (90 days), 240,138 (365 days)
install-on-request: 7,893 (30 days), 21,474 (90 days), 78,615 (365 days)
build-error: 127 (30 days)

[ibearye]

My solution is that: just comment the code of these lines, turn every xspecfem3D commands into executing with mpirun , then everything is ok.

# if [ "$NPROC" -eq 1 ]; then
#   # This is a serial simulation
#   echo
#   echo "  running database generation..."
#   echo
#   ./bin/xgenerate_databases
# else
  # This is a MPI simulation
  echo
  echo "  running database generation on $NPROC processors..."
  echo
  mpirun -np $NPROC ./bin/xgenerate_databases
# fi
# checks exit code
if [[ $? -ne 0 ]]; then exit 1; fi

# runs simulation
# if [ "$NPROC" -eq 1 ]; then
#   # This is a serial simulation
#   echo
#   echo "  running solver..."
#   echo
#   ./bin/xspecfem3D
# else
  # This is a MPI simulation
  echo
  echo "  running solver on $NPROC processors..."
  echo
  mpirun -np $NPROC ./bin/xspecfem3D
# fi

Why errors are thrown while just executing xspecfem3D alone.

[danielpeter]

thanks for the update!

i haven't seen this happen on Intel Macs or Linux with OpenMPI. would you mind providing some more details about your setup? i assume you were compiling the binaries with mpi support, and then the executable fails by just calling, e.g. ./bin/xspecfem3D. what would be your output of

mpif90 -v
mpirun -V

and what would be the trace if you compile by adding the debug flags -g -fbacktrace -ggdb and run the error again? maybe there is some more hint on why this goes haywire.

[ibearye]

The Problem only happened on my apple silicon MacBook pro, and my another intel MacBook works well. I install the requirements by home-brew: open-api, gcc(contains gfortran), and there are the versions bottom:

# mpif90 -v

Using built-in specs.
COLLECT_GCC=/opt/homebrew/bin/gfortran
COLLECT_LTO_WRAPPER=/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../libexec/gcc/aarch64-apple-darwin21/11/lto-wrapper
Target: aarch64-apple-darwin21
Configured with: ../configure --prefix=/opt/homebrew/opt/gcc --libdir=/opt/homebrew/opt/gcc/lib/gcc/11 --disable-nls --enable-checking=release --with-gcc-major-version-only --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-11 --with-gmp=/opt/homebrew/opt/gmp --with-mpfr=/opt/homebrew/opt/mpfr --with-mpc=/opt/homebrew/opt/libmpc --with-isl=/opt/homebrew/opt/isl --with-zstd=/opt/homebrew/opt/zstd --with-pkgversion='Homebrew GCC 11.2.0_3' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --build=aarch64-apple-darwin21 --with-system-zlib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.0 (Homebrew GCC 11.2.0_3) 

# mpirun -V

mpirun (Open MPI) 4.1.3

Report bugs to http://www.open-mpi.org/community/help/

# gcc-11 -v

Using built-in specs.
COLLECT_GCC=gcc-11
COLLECT_LTO_WRAPPER=/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../libexec/gcc/aarch64-apple-darwin21/11/lto-wrapper
Target: aarch64-apple-darwin21
Configured with: ../configure --prefix=/opt/homebrew/opt/gcc --libdir=/opt/homebrew/opt/gcc/lib/gcc/11 --disable-nls --enable-checking=release --with-gcc-major-version-only --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-11 --with-gmp=/opt/homebrew/opt/gmp --with-mpfr=/opt/homebrew/opt/mpfr --with-mpc=/opt/homebrew/opt/libmpc --with-isl=/opt/homebrew/opt/isl --with-zstd=/opt/homebrew/opt/zstd --with-pkgversion='Homebrew GCC 11.2.0_3' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --build=aarch64-apple-darwin21 --with-system-zlib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.0 (Homebrew GCC 11.2.0_3) 

[ibearye]

this comment is about installation:

# ./configure FC=gfortran CC=gcc-11 MPIFC=mpif90  --with-mpi 

## ---------------------------- ##
## setting up compilation flags ##
## ---------------------------- ##
checking build system type... arm-apple-darwin21.4.0
checking host system type... arm-apple-darwin21.4.0
checking whether the Fortran compiler works... yes
checking for Fortran compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU Fortran... yes
checking whether gfortran accepts -g... yes
configure: running /bin/sh ./flags.guess
checking how to get verbose linking output from gfortran... -v
checking for Fortran libraries of gfortran...  
checking for gcc... gcc-11
checking whether the compiler supports GNU C... yes
checking whether gcc-11 accepts -g... yes
checking for gcc-11 option to enable C11 features... none needed
checking for dummy main to link with Fortran libraries... none
checking for Fortran name-mangling scheme... lower case, underscore, no extra underscore
checking for Fortran flag to compile .f90 files... none
checking for Fortran flag needed to accept free-form source... none
checking for Fortran flag to compile preprocessed .F files... none
checking how to define symbols for preprocessed Fortran... -D
checking for Fortran flag to compile preprocessed .F90 files... none
checking for gcc... (cached) gcc-11
checking whether the compiler supports GNU C... (cached) yes
checking whether gcc-11 accepts -g... (cached) yes
checking for gcc-11 option to enable C11 features... (cached) none needed
## ------ ##
## SCOTCH ##
## ------ ##
checking for scotchfarchinit  in -lscotch... no
checking whether Scotch is usable... no, using bundled scotch instead
checking for flex... flex
checking for lex output file root... lex.yy
checking for lex library... not found
configure: WARNING: required lex library not found; giving up on flex
checking for library containing yywrap... no
checking for bison... bison -y
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... -D_THREAD_SAFE
pthread found
SCOTCH is enabled
## ----------------- ##
## module extensions ##
## ----------------- ##
checking Fortran 90 module extension... mod
checking Fortran 90 module inclusion flag... -I
checking Fortran 90 module output flag... -J
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for emmintrin.h... no
checking for xmmintrin.h... no
checking for err.h... yes
checking for g++... g++
checking whether the compiler supports GNU C++... yes
checking whether g++ accepts -g... yes
checking for g++ option to enable C++11 features... none needed
checking how to run the C++ preprocessor... g++ -E
## --- ##
## MPI ##
## --- ##
checking for mpi.h... yes
checking how to run the C preprocessor... gcc-11 -E
## ---------------- ##
## git version info ##
## ---------------- ##
checking for .git/config... yes
configure: building from git repository
## ----------------------------------- ##
## setting up default simulation setup ##
## ----------------------------------- ##
configure: creating ./config.status
config.status: creating Makefile
config.status: creating setup/constants.h
config.status: creating setup/constants_tomography.h
config.status: creating setup/precision.h
config.status: creating setup/config.fh
config.status: creating ./external_libs/scotch/src/Makefile.inc
config.status: creating DATA/Par_file
config.status: creating DATA/CMTSOLUTION
config.status: creating DATA/STATIONS
config.status: creating setup/config.h
config.status: executing bin commands
config.status: executing obj commands
config.status: executing DATA commands
config.status: executing OUTPUT_FILES commands
config.status: executing DATABASES_MPI commands

## --------------- ##
## Specfem3D 3.0.0 ##
## --------------- ##

./configure has completed and set up a default configuration to build.

You may wish to modify the following files before running a simulation:
  DATA/Par_file           Set parameters affecting the simulation.
  DATA/CMTSOLUTION        Set the source parameters before running the solver.
  DATA/STATIONS           Set the receiver stations before running the solver.

after executing ./configure, I execute make, some lines of output that different I found is here:

In file included from graph_io_habo.c:62:
graph_io_habo.c: In function '_SCOTCHgraphGeomLoadHabo':
common.h:116:37: warning: 'realloc' called on pointer '*grafptr.edgetax' with nonzero offset 4 -Wfree-nonheap-object]
  116 | #define memRealloc(ptr,size)        realloc((ptr),((size) | 8))
      |                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
graph_io_habo.c:340:32: note: in expansion of macro 'memRealloc'
  340 |   grafptr->edgetax = ((Gnum *) memRealloc (grafptr->edgetax + 1, grafptr->edgenbr * sizeof (Gnum))) - 1;
      |                                ^~~~~~~~~~

this attachment is all terminal output in compiling.

output.txt

[danielpeter]

great, that will be useful for testing.

regarding the debug flags, after configuration you could add them directly in the SPECFEM3D/Makefile:

FLAGS_CHECK = -g -fbacktrace -ggdb ...

and then recompile. or if preferred adding --enable-debug to the configuration:

./configure --enable-debug ..

for the SCOTCH warnings, that shouldn't cause the above problems with running xmeshfem3D. however, you could try to change the SCOTCH Makefile.inc after the configuration step to match your system. on an older Intel Mac, this was the preferred Makefile.inc:

cp -v external_libs/scotch/src/Make.inc/Makefile.inc.i686_mac_darwin8 external_libs/scotch/src/Makefile.inc