Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyhermes module not found #22

Open
tospines opened this issue Mar 2, 2022 · 30 comments
Open

pyhermes module not found #22

tospines opened this issue Mar 2, 2022 · 30 comments

Comments

@tospines
Copy link
Member

tospines commented Mar 2, 2022

Hello!
Python is having troubles to find the pyhermes module, even though I am importing pyhermes in the build folder, where the pyhermes.cpython-38-x86_64-linux-gnu.so is located. I have tried adding explicitly the build path to the PYTHONPATH variable and to the python sys.path(), but for both cases I get the same error:
>>> import pyhermes
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'pyhermes'

I am installing Hermes in a cluster of computers, within an environment where I have full control, and there was no problem in the installation process.

Any ideas on how to solve this?
Thanks!

@adundovi
Copy link
Contributor

adundovi commented Mar 5, 2022

Hi @tospines, do you use virtualenv to install Hermes on Linux as described here?

@tospines
Copy link
Member Author

tospines commented Mar 7, 2022

Yes, I am totally following the same steps as described there. Curiously I do not get any error or strange warning in the installation step, but then, it does not find the pyhermes package.

@adundovi
Copy link
Contributor

adundovi commented Mar 7, 2022

Here are the steps which should be checked for the Python binding to work:

  1. During the cmake phase one should ensure that the cmake outputs the following (exact versions are not important):
  • Found PythonInterp: ~/.virtualenvs/hermes/bin/python (found version "3.10.2")
  • Found PythonLibs: /usr/lib64/libpython3.10.so (found version "3.10.2")
  • Python: Found!
    -- Version: 3.10/310
    -- Executeable: ~/.virtualenvs/hermes/bin/python
    -- Include: /usr/include/python3.10
    -- Library: /usr/lib64/libpython3.10.so
    -- Site-package directory: ~/.virtualenvs/hermes/lib64/python3.10/site-packages
  1. The build phase should finish successfully, of course, and it has to contain the following line:
  • [100%] Linking CXX shared module pyhermes.cpython-310-x86_64-linux-gnu.so
  1. Testing: with make test one can check all C++ parts of the code where all tests should pass, however, to test the Python binding one can run python interpreter within the build directory where pyhermes.cpython-310-x86_64-linux-gnu.so is present - and there one should be able to import pyhermes:
$ python
Python 3.10.2 (main, Jan 17 2022, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyhermes
>>> dir(pyhermes)
['BremsstrahlungIntegrator', 'CircularWindow', 'DarkMatterIntegrator',  (...), 'units']
  1. If the previous step works, one can finally execute make install which will copy all compiled binaries and resource files to ~/.virtualenvs/hermes/ and one can use pyhermes module from any path as long as the given virtualenv is active.

This has been trying many times on various machines, so please double-check all of these and report back if you have some strange outputs or errors.

@tospines
Copy link
Member Author

tospines commented Mar 7, 2022

I have followed those steps and found a couple of differences with respect what you described:

  • In the cmake step, I do not see any reference to "Found PythonInterp" or "Found PythonLibs",. This is what I got:

    CMAKE_PREFIX_PATH=$HERMES_DIR cmake -DCMAKE_INSTALL_PREFIX=$HERMES_DIR -DENABLE_TESTING=On ..
    -- Build Type: Release
    -- Use --as-needed linker flags!
    -- Use absolute RPATH /home/x_pedde/.virtualenvs/hermes/lib
    -- Downloading data file from heat.gssi.it ~ 130 MB
    -- Extracting data file
    -- FFTW3 with single precision (FFTW3F): Found!
    -- Include: /usr/include
    -- Library: /usr/lib64/libfftw3f.so
    -- found version string 3.49
    -- Found CFITSIO 3.49: /home/x_pedde/cfitsio/cfitsio-3.49/libcfitsio.so
    -- Hermes version: v1.0.1-29-g414f9bf 414f9bf refs/heads/master
    Units dimensionality check enabled.
    -- Python: Found!
    -- Version: 3.6/36
    -- Executeable: /home/x_pedde/.virtualenvs/hermes/bin/python
    -- Include: /software/sse/easybuild/prefix/software/Python/3.6.7-foss-2018a-nsc1/include/python3.6m
    -- Library: /usr/lib64/libpython3.6m.so
    -- Site-package directory: /home/x_pedde/.virtualenvs/hermes/lib/python3.6/site-packages
    -- pybind11 v2.6.3 dev1
    -- Doxygen and Sphinx not found: cannot build documentation
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /proj/dmsearches/users/x_pedde/Hermes/build

Then, after the make -j step, I got the "[100%] Linking CXX shared module pyhermes.cpython-310-x86_64-linux-gnu.so" message as you said. Unfortunately, once I run the make test I get an error at the end:
--The following tests FAILED:
26 - testPixelTimes (Failed)

I am attaching to this message the LastTest.log in case it can help.
LastTest.log
LastTestsFailed.log

Finally, I tried to import pyhermes but it did not work, as earlier.

Should I link the Python interpreter in the cmake step or something like this?

@adundovi
Copy link
Contributor

adundovi commented Mar 7, 2022

26 - testPixelTimes (Failed)

OK, this test can fail as it only measures performance, i.e., pixel calculation times for various integrators to check for performance regressions when introducing new features or changing the code. These times are somehow manually picked by me to match some reasonably recent desktop hardware, but on laptops, these times could be inadequate.

Regarding your issue of loading pyhermes in Python - it looks to me that you have several Python versions installed on your system and with one version your pyhermes module is compiled, while you run the interpreter where you try to load it with the other one.

To understand which Python interpreter you run, please give me the output of: $ python -V. Have you activated a virtual environment before running cmake? You could manually specify the Python paths (PYTHON_INCLUDE_PATH and PYTHON_LIBRARIES) with -D flag in cmake.

@tospines
Copy link
Member Author

tospines commented Mar 7, 2022

Yes, you are right, there are many Python versions installed here. The one I have used for installing and, then, for trying to import pyhermes is Python 3.6.7 (this is the output of python -V).
As I am working in a cluster of computers I first activate an environment and, then, the Hermes environment. Once I am within the HERMES environment, I run the cmake.

@adundovi
Copy link
Contributor

adundovi commented Mar 7, 2022

OK, but you mentioned in your first message a file named pyhermes.cpython-38-x86_64-linux-gnu.so from what I read that Python 3.8 (38) is found during the cmake phase and this will certainly not work with Python 3.6.7. So, either use Python 3.8 within your virtual environment or instruct cmake to find 3.6.7.

@tospines
Copy link
Member Author

tospines commented Mar 7, 2022

Yo were right, it seemed that the python version that I had changed when I entered in my environment. Solved now!
Thank you very much.

@tospines tospines closed this as completed Mar 7, 2022
@fayez0
Copy link

fayez0 commented Mar 20, 2023

Hello!

I am facing the same problem here. I've installed Hermes on Mac OS. I used a virtualenv to install and run it, but I still have the same problem. When I run cmak test, I get this error:

The following tests FAILED:
9 - testIntegrationMethods (Subprocess aborted)
Errors while running CTest
Output from these tests are in: /Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [test] Error 8
(base) fayezabu-ajamieh@Fayezs-MacBook-Pro build % --rerun-failed --output-on-failure
zsh: command not found: --rerun-failed

@adundovi
Copy link
Contributor

Hi @fayez0,
please add the last log file (/Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/Testing/Temporary/LastTest.log) to provide more details.
//Andrej

@fayez0
Copy link

fayez0 commented Mar 22, 2023

Here it is
LastTest.log

@fayez0
Copy link

fayez0 commented Mar 27, 2023

Any idea why I keep getting this error? It seems the code cannot find pyhermes

@fayez0
Copy link

fayez0 commented Apr 3, 2023

Here is the portion of the test that has the error:

9/28 Testing: testIntegrationMethods
9/28 Test: testIntegrationMethods
Command: "/Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/testIntegrationMethods"
Directory: /Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build
"testIntegrationMethods" start time: Apr 03 10:32 IST
Output:

dyld[20643]: symbol not found in flat namespace '_cblas_caxpy'

Test time = 0.01 sec

Test Failed.
"testIntegrationMethods" end time: Apr 03 10:32 IST
"testIntegrationMethods" time elapsed: 00:00:00

@carmeloevoli
Copy link
Member

Could you try to compile it and test it without python first? Just to be sure that the C++ part works properly.
Use -DENABLE_PYTHON=OFF

@fayez0
Copy link

fayez0 commented Apr 3, 2023

I got this:

/Library/Developer/CommandLineTools/usr/bin/make: invalid option -- D
/Library/Developer/CommandLineTools/usr/bin/make: invalid option -- E
Usage: make [options] [target] ...
Options:
-b, -m Ignored for compatibility.
-B, --always-make Unconditionally make all targets.
-C DIRECTORY, --directory=DIRECTORY
Change to DIRECTORY before doing anything.
-d Print lots of debugging information.
--debug[=FLAGS] Print various types of debugging information.
-e, --environment-overrides
Environment variables override makefiles.
-f FILE, --file=FILE, --makefile=FILE
Read FILE as a makefile.
-h, --help Print this message and exit.
-i, --ignore-errors Ignore errors from commands.
-I DIRECTORY, --include-dir=DIRECTORY
Search DIRECTORY for included makefiles.
-j [N], --jobs[=N] Allow N jobs at once; infinite jobs with no arg.
-k, --keep-going Keep going when some targets can't be made.
-l [N], --load-average[=N], --max-load[=N]
Don't start multiple jobs unless load is below N.
-L, --check-symlink-times Use the latest mtime between symlinks and target.
-n, --just-print, --dry-run, --recon
Don't actually run any commands; just print them.
-o FILE, --old-file=FILE, --assume-old=FILE
Consider FILE to be very old and don't remake it.
-p, --print-data-base Print make's internal database.
-q, --question Run no commands; exit status says if up to date.
-r, --no-builtin-rules Disable the built-in implicit rules.
-R, --no-builtin-variables Disable the built-in variable settings.
-s, --silent, --quiet Don't echo commands.
-S, --no-keep-going, --stop
Turns off -k.
-t, --touch Touch targets instead of remaking them.
-v, --version Print the version number of make and exit.
-w, --print-directory Print the current directory.
--no-print-directory Turn off -w, even if it was turned on implicitly.
-W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE
Consider FILE to be infinitely new.
--warn-undefined-variables Warn when an undefined variable is referenced.
-N OPTION, --NeXT-option=OPTION
Turn on value of NeXT OPTION.

This program built for i386-apple-darwin11.3.0
Report bugs to bug-make@gnu.org

By the way, I tried to installed on a remote machine via ssh. It passed all tests, but I still the error:

from pyhermes import *
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'pyhermes'

@carmeloevoli
Copy link
Member

please check it carefully, it cannot be invalid argument:

mkdir build
cd build
cmake  -DENABLE_COVERAGE=OFF -DENABLE_PYTHON=OFF -DENABLE_TESTING=ON .. 

@fayez0
Copy link

fayez0 commented Apr 3, 2023

Same issue

@adundovi
Copy link
Contributor

adundovi commented Apr 3, 2023

dyld[20643]: symbol not found in flat namespace '_cblas_caxpy'

This is an issue, and it seems it is related to GSL and new OS X - look here for a possible solution.

@adundovi adundovi reopened this Apr 3, 2023
@adundovi
Copy link
Contributor

adundovi commented Apr 3, 2023

Maybe reinstalling GLS or BLAS (openblas from homebrew?) and repeating the hermes compilation could fix the issue.

BTW, your issue is unrelated to the first one, for which this thread was created, as in your case the pyhermes module is successfully compiled and all of tests run OK except that one. Hence, in this case, it would be better to open a new ticket.

@fayez0
Copy link

fayez0 commented Apr 4, 2023

This doesn't seem to be the problem: I tried to install hermes on a cluster with Ubuntu 22 using ssh. It installs perfectly and when I run make test, I get:

....
100% tests passed, 0 tests failed out of 28

Total Test time (real) = 12.12 sec

However, I still have the same error.

(hermes) fayez@sudhir-linux-server:~/hermes/build$ python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

from pyhermes import *
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'pyhermes'

@gr-soawnit
Copy link

gr-soawnit commented Nov 23, 2023

I have problem , when I import pyhermes
ImportError: /home/hence/hermes/build/pyhermes.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN6hermes11nside2orderEj

make -j
[100%] Built target pyhermes
-- Build Type: Release
-- Use --as-needed linker flags!
-- Use absolute RPATH /home/hence/hermes/lib
-- Downloading data file from heat.gssi.it ~ 200 MB
-- Extracting data file
-- Found PythonInterp: /home/hence/anaconda3/envs/hermes/bin/python (found version "3.11.5")
-- FFTW3F header => /home/hence/anaconda3/envs/hermes/include
-- FFTW3F libs => /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so
-- FFTW3 with single precision (FFTW3F): Found!
-- Include: /home/hence/anaconda3/envs/hermes/include
-- Library: /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so
-- found version string 4.3
-- Found CFITSIO 4.3: /home/hence/anaconda3/envs/hermes/lib/libcfitsio.so
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Parallel hdf5: FALSE
-- Hermes version: v1.0.1-96-g165cd2ac 165cd2a refs/heads/master
Enabling coverage report via $make coverage
Units dimensionality check enabled.
-- Python: Found!
-- Version: 3.11/311
-- Executeable: /home/hence/anaconda3/envs/hermes/bin/python
-- Include: /home/hence/anaconda3/envs/hermes/include/python3.11
-- Library: /home/hence/anaconda3/envs/hermes/lib/libpython3.11.so
-- Site-package directory: /home/hence/anaconda3/envs/hermes/lib/python3.11/site-packages
-- pybind11 v2.11.0 dev1
-- Found Doxygen and Sphinx to build documentation
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hence/hermes/build
(hermes) hence@DESKTOP-M6C1FJR:~/hermes/build$ CMAKE_PREFIX_PATH=$HERMES_DIR cmake -DCMAKE_INSTALL_PREFIX=$HERMES_DIR -DENABLE_TESTING=On ..
-- Build Type: Release
-- Use --as-needed linker flags!
-- Use absolute RPATH /home/hence/hermes/lib
-- Downloading data file from heat.gssi.it ~ 200 MB
-- Extracting data file
-- FFTW3F header => /home/hence/anaconda3/envs/hermes/include
-- FFTW3F libs => /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so
-- FFTW3 with single precision (FFTW3F): Found!
-- Include: /home/hence/anaconda3/envs/hermes/include
-- Library: /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so
-- found version string 4.3
-- Found CFITSIO 4.3: /home/hence/anaconda3/envs/hermes/lib/libcfitsio.so
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Parallel hdf5: FALSE
-- Hermes version: v1.0.1-96-g165cd2ac 165cd2a refs/heads/master
Enabling coverage report via $make coverage
Units dimensionality check enabled.
-- Python: Found!
-- Version: 3.11/311
-- Executeable: /home/hence/anaconda3/envs/hermes/bin/python
-- Include: /home/hence/anaconda3/envs/hermes/include/python3.11
-- Library: /home/hence/anaconda3/envs/hermes/lib/libpython3.11.so
-- Site-package directory: /home/hence/anaconda3/envs/hermes/lib/python3.11/site-packages
-- pybind11 v2.11.0 dev1
-- Found Doxygen and Sphinx to build documentation
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hence/hermes/build

@adundovi
Copy link
Contributor

Hi @gr-soawnit, which version/distribution of Linux do you use?

@gr-soawnit
Copy link

gr-soawnit commented Dec 15, 2023 via email

@gr-soawnit
Copy link

I try to fix the problem. Now I got
[ 5%] Built target cparamlib
[ 8%] Built target kiss
[ 46%] Built target hermes
[ 59%] Built target pyhermes
[ 60%] Built target gtest
[ 61%] Built target gtest_main
[ 62%] Built target testPicard3D
[ 63%] Built target testHdf5Reader
[ 65%] Built target testUnits
[ 66%] Built target testCommon
[ 68%] Built target testNumericalIntegration
[ 70%] Built target testCacheTools
[ 71%] Built target testVector3
[ 73%] Built target testHEALPix
[ 74%] Built target testIntegrationMethods
[ 76%] Built target testSkymap
[ 77%] Built target testFITS
[ 78%] Built target testCosmicRays
[ 79%] Built target testYMW16
[ 81%] Built target testDispersionMeasureIntegrator
[ 82%] Built target testRotationMeasureIntegrator
[ 83%] Built target testSynchroIntegrator
[ 84%] Built target testFreeFreeIntegrator
[ 85%] Built target testMFieldModels
[ 86%] Built target testSynchroAbsorptionIntegrator
[ 88%] Built target testInteractions
[ 89%] Built target testRingModel
[ 90%] Built target testPiZeroIntegrator
[ 91%] Built target testBremsstrahlungIntegrator
[ 92%] Built target testPhotonField
[ 93%] Built target testInverseComptonIntegrator
[ 94%] Built target testBreitWheeler
[ 95%] Built target testDarkMatter
[ 97%] Built target testPixelTimes
[100%] Built target test_uuid
Install the project...
-- Install configuration: "Release"
-- Installing: /home/codex/anaconda3/envs/hermes/lib/python3.12/site-packages/pyhermes.cpython-312-x86_64-linux-gnu.so
-- Set runtime path of "/home/codex/anaconda3/envs/hermes/lib/python3.12/site-packages/pyhermes.cpython-312-x86_64-linux-gnu.so" to ""
-- Installing: /lib/libhermes.so
CMake Error at cmake_install.cmake:80 (file):
file INSTALL cannot copy file "/home/codex/hermes/build/libhermes.so" to
"/lib/libhermes.so": Permission denied.

I cann't do with sudo make install
sudo: make: command not found

@adundovi
Copy link
Contributor

adundovi commented Jan 11, 2024

Hi, @gr-soawnit, this error means that you didn't specified a custom install path and the default system one (/lib/) is used for which you need the root permission to copy into.

-- Installing: /lib/libhermes.so
CMake Error at cmake_install.cmake:80 (file):
file INSTALL cannot copy file "/home/codex/hermes/build/libhermes.so" to
"/lib/libhermes.so": Permission denied.

You can read easily this output: "...trying to install /lib/libhermes.so by copying the file from /home/codex/hermes/build/libhermes.so... err, unfortunately, didn't work: permission denied." So, either you can run cmake install with sudo in front to run it with the admin permission, which will certainly copy it to /lib. Or you can repeat the step cmake .. where you should use -DCMAKE_INSTALL_PREFIX=$HERMES_DIR. And $HERMES_DIR is an environment variable where you would like to install Hermes in some directory where you have permission to copy files into such as, HERMES_DIR=$HOME"/.virtualenvs/hermes". Please, see "Install on GNU/Linux" in INSTALL.

@gr-soawnit
Copy link

Hi, @adundovi , I try to solve this follow your advice but it already still have same problem ImportError: /home/hence/hermes/build/pyhermes.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN6hermes11nside2orderEj

now, I install in ubuntu 22.24, another partition, it got back to problem "ModuleNotFoundError: No module named 'pyhermes'"
I follow to your installation, I got

CMAKE_PREFIX_PATH=$HERMES_DIR cmake -DCMAKE_INSTALL_PREFIX=$HERMES_DIR -DENABLE_COVERAGE=ON -DENABLE_PYTHON=ON -DENABLE_TESTING=ON ..

-- Build Type: Release
-- Use --as-needed linker flags!
-- Use absolute RPATH /home/codex/.virtualenvs/hermes/lib
-- Downloading data file from heat.gssi.it ~ 200 MB
-- Extracting data file
-- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter
-- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- FFTW3F header => /usr/include
-- FFTW3F libs => /usr/lib/x86_64-linux-gnu/libfftw3f.so
-- FFTW3 with single precision (FFTW3F): Found!
-- Include: /usr/include
-- Library: /usr/lib/x86_64-linux-gnu/libfftw3f.so
-- found version string 4.0
-- Found CFITSIO 4.0: /usr/lib/x86_64-linux-gnu/libcfitsio.so
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS) (found version "")
-- Hermes version: v1.0.1-97-g16299515 1629951 refs/heads/master
Enabling coverage report via $make coverage
Units dimensionality check enabled.
-- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter
-- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- Found Doxygen and Sphinx to build documentation
-- Configuring done
-- Generating done
-- Build files have been written to: /home/codex/hermes/build

@gr-soawnit
Copy link

I install python 3.10.12 for new installation

@adundovi
Copy link
Contributor

ModuleNotFoundError: No module named 'pyhermes'" could mean two things: either no python-dev package was found during the cmake procedure, or there is a mistake in Python paths. In both cases Python will not be able to find pyhermes.

Do you know how to use Docker? I can prepare a docker image with jupyter and HERMES which should straightforward to install?

@gr-soawnit
Copy link

gr-soawnit commented Jan 11, 2024

@adundovi I already have python-dev but don't sure, this cann't find path

Docker image, It okay. I have used it before, but I'm not familiar with it. it will be helpful.

@adundovi
Copy link
Contributor

@gr-soawnit, I updated the code with two things:

  1. I have cleaned up and updated the CMake code for finding Python, so I hope it will work better with newer versions of various software stacks;
  2. Now, there is a Dockerfile with which you can easily build and run a docker image with Jupyter and HERMES installed:
docker build -t jupyter-pyhermes .
docker run --name jupyter -it --rm -p 8888:8888 -v $HOME/jupyter-work:/home/jovyan/work localhost/jupyter-pyhermes:latest

See the bottom of INSTALL for more information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants