Boost.Python linking on linux and python installed from source

[tomaszkacprzak]

As before, I am running Scientific Linux SL release 5.4 (Boron) with python 2.7.1 and mkl 11.1 046 and no root privileges (UCL Starlink).

I installed python2.7 from source to a local folder.
My boost installation is

./bootstrap.sh --with-python=python2.7 --with-python-root=/home/kacprzak/local/lib/python2.7/
./b2 link=shared
./b2 --prefix=/home/kacprzak/local link=shared install

When I check the python version against which boost was installed, as instructed in INSTALL, I don't see the python version number:

bash-3.2$ ldd ~/local/lib/libboost_python.so
        linux-vdso.so.1 =>  (0x00007fff3d0f7000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00002adced1cb000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002adced3cf000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002adced5ea000)
        librt.so.1 => /lib64/librt.so.1 (0x00002adced7ee000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002adced9f8000)
        libm.so.6 => /lib64/libm.so.6 (0x00002adcedcf8000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002adcedf7b000)
        libc.so.6 => /lib64/libc.so.6 (0x00002adcee18a000)
        /lib64/ld-linux-x86-64.so.2 (0x000000365a800000)

I run GalSim scons with

CXX = 'icpc'
PYTHON = 'python2.7'
PREFIX = '/home/kacprzak/code/GalSim_install'
PYPREFIX = '/home/kacprzak/local/lib/python2.7/'
TMV_DIR = '/home/kacprzak/local/tmv'
FFTW_DIR = '/home/kacprzak/local/fftw3'
IMPORT_PREFIX = False
CACHE_LIB = False

And I get an error

Unable to build a python loadable module with Boost.Python

Rachel suggested that this may be an issue with static/dynamic installation of python from source.

This is a relevant fragment of config.log.

scons: Configure: Checking if we can build against Boost.Python...
.sconf_temp/conftest_14.cpp <-
  |
  |#include "boost/python.hpp"
  |
  |int check_bp_run() { return 23; }
  |
  |BOOST_PYTHON_MODULE(check_bp) {
  |    boost::python::def("run",&check_bp_run);
  |}
  |
icpc -o .sconf_temp/conftest_14.o -c -O2 -vec-report0 -g -Wall -Werror -wd279,383,810,981 -wd1572 -wd2259 -Iinclude -I/home/kacprzak/local/tmv/include -I/home/kacprzak/local/fftw3/include -I/home/kacprzak/local/include/python2.7 -I/home/kacprzak/local/lib/python2.7/site-packages/numpy/core/include .sconf_temp/conftest_14.cpp
.sconf_temp/conftest_14.cpp(4): remark #1418: external function definition with no prior declaration
  int check_bp_run() { return 23; }
      ^

.sconf_temp/conftest_14.cpp(6): remark #1419: external declaration in primary source file
  BOOST_PYTHON_MODULE(check_bp) {
  ^

.sconf_temp/conftest_14.cpp(6): remark #1418: external function definition with no prior declaration
  BOOST_PYTHON_MODULE(check_bp) {
  ^

.sconf_temp/conftest_15.cpp <-
  |
  |#include "boost/python.hpp"
  |
  |int check_bp_run() { return 23; }
  |
  |BOOST_PYTHON_MODULE(check_bp) {
  |    boost::python::def("run",&check_bp_run);
  |}
  |
icpc -o .sconf_temp/conftest_15.os -c -O2 -vec-report0 -g -Wall -Werror -wd279,383,810,981 -wd1572 -wd2259 -fPIC -Iinclude -I/home/kacprzak/local/tmv/include -I/home/kacprzak/local/fftw3/include -I/home/kacprzak/local/include/python2.7 -I/home/kacprzak/local/lib/python2.7/site-packages/numpy/core/include .sconf_temp/conftest_15.cpp
.sconf_temp/conftest_15.cpp(4): remark #1418: external function definition with no prior declaration
  int check_bp_run() { return 23; }
      ^

.sconf_temp/conftest_15.cpp(6): remark #1419: external declaration in primary source file
  BOOST_PYTHON_MODULE(check_bp) {
  ^

.sconf_temp/conftest_15.cpp(6): remark #1418: external function definition with no prior declaration
  BOOST_PYTHON_MODULE(check_bp) {
  ^

icpc -o .sconf_temp/conftest_15_mod/check_bp.so -openmp -shared .sconf_temp/conftest_15.os -Llib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread
.sconf_temp/conftest_16 <-
  |import sys; sys.path.append('.sconf_temp/conftest_15_mod'); import check_bp; print check_bp.run()
python2.7 < .sconf_temp/conftest_16 > .sconf_temp/conftest_16.out
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: .sconf_temp/conftest_15_mod/check_bp.so: undefined symbol: _ZNK5boost6python7objects21py_function_impl_base9max_arityEv
.sconf_temp/conftest_17.cpp <-
  |
  |#include "boost/python.hpp"
  |
  |int check_bp_run() { return 23; }
  |
  |BOOST_PYTHON_MODULE(check_bp) {
  |    boost::python::def("run",&check_bp_run);
  |}
  |
icpc -o .sconf_temp/conftest_17.os -c -O2 -vec-report0 -g -Wall -Werror -wd279,383,810,981 -wd1572 -wd2259 -fPIC -Iinclude -I/home/kacprzak/local/tmv/include -I/home/kacprzak/local/fftw3/include -I/home/kacprzak/local/include/python2.7 -I/home/kacprzak/local/lib/python2.7/site-packages/numpy/core/include .sconf_temp/conftest_17.cpp
.sconf_temp/conftest_17.cpp(4): remark #1418: external function definition with no prior declaration
  int check_bp_run() { return 23; }
      ^

.sconf_temp/conftest_17.cpp(6): remark #1419: external declaration in primary source file
  BOOST_PYTHON_MODULE(check_bp) {
  ^

.sconf_temp/conftest_17.cpp(6): remark #1418: external function definition with no prior declaration
  BOOST_PYTHON_MODULE(check_bp) {
  ^

icpc -o .sconf_temp/conftest_17_mod/check_bp.so -openmp -shared .sconf_temp/conftest_17.os -Llib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -lboost_python -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread
.sconf_temp/conftest_18 <-
  |import sys; sys.path.append('.sconf_temp/conftest_17_mod'); import check_bp; print check_bp.run()
python2.7 < .sconf_temp/conftest_18 > .sconf_temp/conftest_18.out
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /usr/lib64/libboost_python.so.2: undefined symbol: PyUnicodeUCS4_FromEncodedObject
.sconf_temp/conftest_19.cpp <-
  |
  |#include "boost/python.hpp"
  |
  |int check_bp_run() { return 23; }
  |
  |BOOST_PYTHON_MODULE(check_bp) {
  |    boost::python::def("run",&check_bp_run);
  |}
  |
icpc -o .sconf_temp/conftest_19.os -c -O2 -vec-report0 -g -Wall -Werror -wd279,383,810,981 -wd1572 -wd2259 -fPIC -Iinclude -I/home/kacprzak/local/tmv/include -I/home/kacprzak/local/fftw3/include -I/home/kacprzak/local/include/python2.7 -I/home/kacprzak/local/lib/python2.7/site-packages/numpy/core/include .sconf_temp/conftest_19.cpp
.sconf_temp/conftest_19.cpp(4): remark #1418: external function definition with no prior declaration
  int check_bp_run() { return 23; }
      ^

.sconf_temp/conftest_19.cpp(6): remark #1419: external declaration in primary source file
  BOOST_PYTHON_MODULE(check_bp) {
  ^

.sconf_temp/conftest_19.cpp(6): remark #1418: external function definition with no prior declaration
  BOOST_PYTHON_MODULE(check_bp) {
  ^

icpc -o .sconf_temp/conftest_19_mod/check_bp.so -openmp -shared .sconf_temp/conftest_19.os -Llib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -lboost_python-mt -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread
ld: cannot find -lboost_python-mt

My LD_LIBRARY_PATH is updated:

bash-3.2$ echo $LD_LIBRARY_PATH
/home/kacprzak/local/lib:/home/kacprzak/code/GalSim/lib:/usr/lib64/atlas: ...

[barnabytprowe]

Hi Tomek,

The line: ImportError: /usr/lib64/libboost_python.so.2: undefined symbol: PyUnicodeUCS4_FromEncodedObject

...suggests that you are not linking to the Boost directory you installed Boost into yourself. I noticed you didn't set a BOOST_DIR= flag... It looks like there is already a Boost install in /usr/lib64/ that will be incompatible with your self-installed Python.

[one thought: @rmjarvis : off the top of your head is it possible to tell scons to ignore certain directories? Or at least very commonly used ones, something along the lines of: http://stackoverflow.com/questions/2988779/gcc-how-to-ignore-standard-include-paths ? Might be nice for these common cases we get where people don't have root access... Might be worth putting in an FAQ for gcc users at least, what do you think? (I'm happy to do it)]

Another thought: in case of further icpc/gcc conflicts, once you've got the correct Boost actually being linked, if that doesn't work you should reinstall Boost using the icpc too... This can be done with the options toolset=intel or toolset=intel-linux at the ./bootstrap stage.

Barnaby Rowe
Postdoctoral Research Associate

Department of Physics & Astronomy
University College London
Gower Street
London WC1E 6BT
United Kingdom

On 20 Sep 2012, at 17:58, Tomasz Kacprzak wrote:

ImportError: /usr/lib64/libboost_python.so.2: undefined symbol: PyUnicodeUCS4_FromEncodedObject

[rmjarvis]

Can you also check which python runs on your command line when you type python2.7? Two ways to check:

which python2.7
python2.7 -c "import sys; print sys.executable"

Maybe do both just to be sure.

[rmjarvis]

off the top of your head is it possible to tell scons to ignore certain directories?

You can do what the stackoverflow site says by doing scons EXTRA_FLAGS='-nostdinc -nostdinc++'. But I don't think that would help in this case. The problem is at the linking step. And we don't want to ignore /usr/lib64, since that has other libraries that do need to be linked. So I think any attempt to ignore this directory will lead to more problems. Better to make sure we're linking to the right boost and boost links to the right python, etc.

[tomaszkacprzak]

Fortunately using BOOST_DIR cased the problem to disappear, well spotted Barney!

Now I am getting the same as before, described in Issue #261.

Python is:

bash-3.2$ which python2.7
/home/kacprzak/local/bin/python2.7
bash-3.2$ python2.7 -c "import sys; print sys.executable"
/home/kacprzak/local/bin/python2.7

[rmjarvis]

Fortunately using BOOST_DIR cased the problem to disappear, well spotted Barney!
Now I am getting the same as before, described in Issue #261.

OK. Closing this issue then. Moving discussion to #261.