Install issue: "cannot enable executable stack"

[lukashergt]

I am having issues with fresh PolyChord installs on my local machines (Manjaro Linux). This might be related to #119 or #121, but it could be down to me having messed up my system.

Anyone experiencing similar issues or anyone who has encountered something similar before?

Details

Make command

I started by freshly git cloning the rep and running make with the following output:

(Click here)

make -C /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/src/polychord /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/lib/libchord.a
make[1]: Entering directory '/home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/src/polychord'
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c utils.F90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c abort.F90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c array_utils.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c settings.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c calculate.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c mpi_utils.F90
mpi_utils.F90:627:12:

  627 |             empty_buffer,                &! not sending anything
      |            1
......
  655 |             live_point,                  &! live point being sent
      |            2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
mpi_utils.F90:581:12:

  581 |             logL,                        &!
      |            1
......
  655 |             live_point,                  &! live point being sent
      |            2
Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)
mpi_utils.F90:517:12:

  517 |             logL,                        &!
      |            1
......
  680 |             live_point,                  &! live point recieved
      |            2
Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)
mpi_utils.F90:527:12:

  527 |             epoch,                       &!
      |            1
......
  680 |             live_point,                  &! live point recieved
      |            2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
mpi_utils.F90:275:12:

  275 |             doubles,                     &!broadcast buffer
      |            1
......
  291 |             integers,                    &!broadcast buffer
      |            2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)).
mpi_utils.F90:240:12:

  240 |             intgr_local,                 &!send buffer
      |            1
......
  258 |             db_local,                    &!send buffer
      |            2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
mpi_utils.F90:241:12:

  241 |             intgr,                       &!recieve buffer
      |            1
......
  259 |             db,                          &!recieve buffer
      |            2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c random_utils.F90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c chordal_sampling.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c run_time_info.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c clustering.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c params.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c priors.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c read_write.F90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c feedback.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c generate.F90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c ini.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c nelder_mead.f90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c maximiser.F90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c nested_sampling.F90
mpifort -ffree-line-length-none -cpp -fPIC -fno-stack-arrays  -fallow-argument-mismatch -Ofast -DMPI -c interfaces.F90
mpicxx -std=c++11 -fPIC -Ofast -DUSE_MPI -c c_interface.cpp
ar rv /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/lib/libchord.a abort.o array_utils.o calculate.o chordal_sampling.o clustering.o feedback.o generate.o ini.o interfaces.o maximiser.o mpi_utils.o nelder_mead.o nested_sampling.o params.o priors.o random_utils.o read_write.o run_time_info.o settings.o utils.o c_interface.o
ar: creating /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/lib/libchord.a
a - abort.o
a - array_utils.o
a - calculate.o
a - chordal_sampling.o
a - clustering.o
a - feedback.o
a - generate.o
a - ini.o
a - interfaces.o
a - maximiser.o
a - mpi_utils.o
a - nelder_mead.o
a - nested_sampling.o
a - params.o
a - priors.o
a - random_utils.o
a - read_write.o
a - run_time_info.o
a - settings.o
a - utils.o
a - c_interface.o
make[1]: Leaving directory '/home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/src/polychord'
make -C /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/src/polychord /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/lib/libchord.so
make[1]: Entering directory '/home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/src/polychord'
mpifort -shared abort.o array_utils.o calculate.o chordal_sampling.o clustering.o feedback.o generate.o ini.o interfaces.o maximiser.o mpi_utils.o nelder_mead.o nested_sampling.o params.o priors.o random_utils.o read_write.o run_time_info.o settings.o utils.o c_interface.o -o /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/lib/libchord.so -lstdc++
/usr/bin/ld: warning: interfaces.o: requires executable stack (because the .note.GNU-stack section is executable)
make[1]: Leaving directory '/home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/src/polychord'

• Sidenote: The various mismatch warnings will cause compilation to fail with DEBUG=1 because of the -pedantic flag, which I tried out, but to no avail so far.
• Possibly more important. Note the warning in the second last line might be relevant:

  /usr/bin/ld: warning: interfaces.o: requires executable stack (because the .note.GNU-stack section is executable)
  

• I have initially used gcc 14.2.1, but I also tried gcc 10.5.0---same outcome.

Pip install

I run pip install . with the following output:

(Click here)

Processing /home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy in /home/hergtl/.virtualenvs/tflatest_3-11-9_env/lib/python3.11/site-packages (from pypolychord==1.22.1) (1.24.3)
Requirement already satisfied: scipy in /home/hergtl/.virtualenvs/tflatest_3-11-9_env/lib/python3.11/site-packages (from pypolychord==1.22.1) (1.14.1)
Building wheels for collected packages: pypolychord
  Building wheel for pypolychord (pyproject.toml) ... done
  Created wheel for pypolychord: filename=pypolychord-1.22.1-cp311-cp311-linux_x86_64.whl size=356598 sha256=2c54bb01c347832a41237252a7a7f7ccdcb5e810158b24565aab616c1ed2e461
  Stored in directory: /tmp/pip-ephem-wheel-cache-w0depkpz/wheels/1a/8e/e5/071454c3312e72cc6151b49579a649f881f5d389831aa7cd34
Successfully built pypolychord
Installing collected packages: pypolychord
Successfully installed pypolychord-1.22.1

However, when I then run python run_pypolychord.py, I get the following error:

Traceback (most recent call last):
  File "/home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/run_pypolychord.py", line 2, in <module>
    import pypolychord
  File "/home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/pypolychord/__init__.py", line 3, in <module>
    from pypolychord.polychord import run_polychord, run
  File "/home/hergtl/Documents/Projects/PolyChordPrj/PolyChordLite/pypolychord/polychord.py", line 4, in <module>
    import _pypolychord
ImportError: libchord.so: cannot enable executable stack as shared object requires: Invalid argument

Sidenote: I did try python setup.py install as well, ultimately producing the same error but with additional warnings during install.

Attempt execstack

Following the simple suggestion to run sudo execstack -c lib/libchord.so, I can now import pypolychord without an error, however when trying to run python run_pypolychord.py, it now crashes with a segfault when trying to generate live points:

(Click here)

PolyChord: MPI is already initilised, not initialising, and will not finalize

PolyChord: Next Generation Nested Sampling
copyright: Will Handley, Mike Hobson & Anthony Lasenby
  version: 1.22.1
  release: 10th Jan 2024
    email: wh260@mrao.cam.ac.uk

Run Settings
nlive    :     200
nDims    :       4
nDerived :       1
Doing Clustering
Synchronous parallelisation
Generating equally weighted posteriors
Generating weighted posteriors
Clustering on posteriors
Writing a resume file to chains/gaussian.resume

generating live points

[lukas-amd3950x:254330:0:254330] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7ffddde81704)
==== backtrace (tid: 254330) ====
 0 0x000000000004ca22 ucs_event_set_fd_get()  ???:0
 1 0x000000000004cbed ucs_event_set_fd_get()  ???:0
 2 0x000000000003dcd0 __sigaction()  ???:0
=================================
Segmentation fault (core dumped)

Further checks

I've tried to find out a bit more about the executable stack warning. readelf -SW src/polychord/interfaces.o | grep stack shows that interfaces.o does indeed have an X in it, so does maximiser.o. But I am a bit out of my element here and welcome any suggestions/insight.

[williamjameshandley]

Many thanks for the detailed issue, which has also been raised privately by @Mauropieroni (I don't know if he's made any further progress). I'm looking into this, as it also fails on my arch linux system.

[Mauropieroni]

Unfortunately, not much more than that.
The other thing I found out is that the issue comes with the update 2.41 of glibc.
I tried playing with the flags you pass to the compiler, but nothing worked.

[williamjameshandley]

OK, an hours marathon vibing with o3-mini high eventually revealed the problem to be a combination of needing to add some linker flags, and a re-write of the interfaces so that it does not define functions within a function's contains section.

@lukashergt and @Mauropieroni, could you check out #125 and see if that fixes your systems?

[Mauropieroni]

@williamjameshandley
I copied locally the changes you did in #125 and it does work.
Everything seems to work also with MPI.
Thanks Will!

[lukashergt]

Wow, thanks for the quick reply! This seems to be going in the right direction, further comments in #125.

[dstaicova]

Hi guys, I included the Makefile_gnu in Makefile and copied the new interfaces.F90 but it still gives the same error. I tried manually compiling the *.F90 (as suggested by some llm in lmarena) but I keep on trying to fix the order, which is quite annoying. Can we get a minimally working update for Linux? I'm on Manjaro, getting the same error and so basically I'm without polychord after the system update.
Edit: Problem fixed by changing the Makefile in /src/polychord to:

$(LIB_DIR)/libchord.so: $(OBJECTS) $(CXX) -shared $^ -o $@ $(LDLIBS) $(RPATH) $(LDFLAGS)