Install on a mac?

[keflavich]

I'm attempting to install pybdsf on a mac using anaconda. My attempts are stymied at this error:

ld: library not found for -lboost_python3-mt

my conda environment is:

conda list
# packages in environment at /Users/adam/miniconda3:
#
# Name                    Version                   Build  Channel
appnope                   0.1.0                    py37_0
asn1crypto                0.24.0                   py37_0
astropy                   3.2.1            py37h01d97ff_0    astropy
atomicwrites              1.3.0                    py37_1
attrs                     19.1.0                   py37_1
backcall                  0.1.0                    py37_0
blas                      1.0                         mkl
bleach                    3.1.0                    py37_0
boost                     1.67.0                   py37_4
bzip2                     1.0.6                h1de35cc_5
ca-certificates           2019.5.15                     0    anaconda
certifi                   2019.6.16                py37_0    anaconda
cffi                      1.12.3           py37hb5b8e2f_0
chardet                   3.0.4                    py37_1
cloog                     0.18.0                        0    anaconda
conda                     4.7.5                    py37_0    anaconda
conda-package-handling    1.3.10                   py37_0
cryptography              2.7              py37ha12b0ac_0
cycler                    0.10.0                   py37_0
dbus                      1.13.6               h90a0687_0
decorator                 4.4.0                    py37_1
defusedxml                0.6.0                      py_0
entrypoints               0.3                      py37_0
expat                     2.2.6                h0a44026_0
freetype                  2.9.1                hb4e5f40_0
gcc                       4.8.5                         8    anaconda
gettext                   0.19.8.1             h15daf44_3
glib                      2.56.2               hd9629dc_0
gmp                       6.1.2                hb37e062_1    anaconda
icu                       58.2                 h4b95b61_1
idna                      2.8                      py37_0
importlib_metadata        0.17                     py37_1
intel-openmp              2019.4                      233
ipykernel                 5.1.1            py37h39e3cac_0
ipython                   7.6.0            py37h39e3cac_0
ipython_genutils          0.2.0                    py37_0
ipywidgets                7.4.2                    py37_0
isl                       0.12.2                        1    anaconda
jedi                      0.13.3                   py37_0
jinja2                    2.10.1                   py37_0
jpeg                      9b                   he5867d9_2
jsonschema                3.0.1                    py37_0
jupyter                   1.0.0                    py37_7
jupyter_client            5.2.4                    py37_0
jupyter_console           6.0.0                    py37_0
jupyter_core              4.5.0                      py_0
jupyterlab                0.35.5           py37hf63ae98_0
jupyterlab_server         0.2.0                    py37_0
kiwisolver                1.1.0            py37h0a44026_0
libarchive                3.3.3                h786848e_5
libboost                  1.67.0               hebc422b_4
libcxx                    4.0.1                hcfea43d_1
libcxxabi                 4.0.1                hcfea43d_1
libedit                   3.1.20181209         hb402a30_0
libffi                    3.2.1                h475c297_4
libgfortran               3.0.1                h93005f0_2
libiconv                  1.15                 hdd342a3_7
libpng                    1.6.37               ha441bb4_0
libsodium                 1.0.16               h3efe00b_0
libxml2                   2.9.9                hf6e021a_1
lz4-c                     1.8.1.2              h1de35cc_0
lzo                       2.10                 h362108e_2
markupsafe                1.1.1            py37h1de35cc_0
matplotlib                3.1.0            py37h54f8f79_0
mistune                   0.8.4            py37h1de35cc_0
mkl                       2019.4                      233
mkl_fft                   1.0.12           py37h5e564d8_0
mkl_random                1.0.2            py37h27c97d8_0
more-itertools            7.0.0                    py37_0
mpc                       1.1.0                h6ef4df4_1    anaconda
mpfr                      4.0.1                h3018a27_3    anaconda
nbconvert                 5.5.0                      py_0
nbformat                  4.4.0                    py37_0
ncurses                   6.1                  h0a44026_1
notebook                  5.7.8                    py37_0
numpy                     1.16.4           py37hacdab7b_0
numpy-base                1.16.4           py37h6575580_0
openssl                   1.1.1                h1de35cc_0    anaconda
packaging                 19.0                     py37_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py37_1
parso                     0.5.0                      py_0
pcre                      8.43                 h0a44026_0
pexpect                   4.7.0                    py37_0
pickleshare               0.7.5                    py37_0
pip                       19.1.1                   py37_0
pluggy                    0.12.0                     py_0
prometheus_client         0.7.1                      py_0
prompt_toolkit            2.0.9                    py37_0
psutil                    5.6.3            py37h1de35cc_0
ptyprocess                0.6.0                    py37_0
py                        1.8.0                    py37_0
py-boost                  1.67.0           py37h6440ff4_4
pycosat                   0.6.3            py37h1de35cc_0
pycparser                 2.19                     py37_0
pygments                  2.4.2                      py_0
pyopenssl                 19.0.0                   py37_0
pyparsing                 2.4.0                      py_0
pyqt                      5.9.2            py37h655552a_2
pyrsistent                0.14.11          py37h1de35cc_0
pysocks                   1.7.0                    py37_0
pytest                    5.0.0                    py37_0
pytest-arraydiff          0.3                        py_0    astropy
pytest-astropy            0.5.0                      py_0    astropy
pytest-doctestplus        0.3.0                      py_0    astropy
pytest-openfiles          0.3.1                      py_0    astropy
pytest-remotedata         0.3.1                      py_0    astropy
python                    3.7.3                h359304d_0
python-dateutil           2.8.0                    py37_0
python-libarchive-c       2.8                      py37_8
pytz                      2019.1                     py_0
pyzmq                     18.0.0           py37h0a44026_0
qt                        5.9.7                h468cd18_1
qtconsole                 4.5.1                      py_0
readline                  7.0                  h1de35cc_5
requests                  2.22.0                   py37_0
ruamel_yaml               0.15.46          py37h1de35cc_0
scipy                     1.2.1            py37h1410ff5_0
send2trash                1.5.0                    py37_0
setuptools                41.0.1                   py37_0
sip                       4.19.8           py37h0a44026_0
six                       1.12.0                   py37_0
sqlite                    3.28.0               ha441bb4_0
terminado                 0.8.2                    py37_0
testpath                  0.4.2                    py37_0
tk                        8.6.8                ha441bb4_0
tornado                   6.0.3            py37h1de35cc_0
tqdm                      4.32.1                     py_0
traitlets                 4.3.2                    py37_0
urllib3                   1.24.2                   py37_0
wcwidth                   0.1.7                    py37_0
webencodings              0.5.1                    py37_1
wheel                     0.33.4                   py37_0
widgetsnbextension        3.4.2                    py37_0
xz                        5.2.4                h1de35cc_4
yaml                      0.1.7                hc338f04_2
zeromq                    4.3.1                h0a44026_3
zipp                      0.5.1                      py_0
zlib                      1.2.11               h1de35cc_3
zstd                      1.3.7                h5bba6e5_0

most importantly:

boost                     1.67.0                   py37_4
libboost                  1.67.0               hebc422b_4
py-boost                  1.67.0           py37h6440ff4_4

Any ideas? Do I just need an older version of boost / py-boost / libboost ? Are there any reports of anyone successfully building pybdsf w/anaconda?

[darafferty]

Unfortunately, the current setup.py doesn't seem to handle this case well. I did get it to work on my Mac with the following changes (ideally these should be incorporated into setup.py at some point):

--- a/setup.py
+++ b/setup.py
@@ -103,7 +103,7 @@ def find_boost():
         if sys.version_info[0] == 2:
             boost_python = "boost_python-mt"
         else:
-            boost_python = "boost_python3-mt"
+            boost_python = "boost_python37"
     return boost_python
 
 
@@ -128,7 +128,7 @@ def find_boost_numpy():
         if sys.version_info[0] == 2:
             boost_numpy = "boost_numpy-mt"
         else:
-            boost_numpy = "boost_numpy3-mt"
+            boost_numpy = "boost_numpy37"
 
         if not find_library(boost_numpy):
             return None
@@ -173,8 +173,8 @@ def main():
             "src/c++/num_util/num_util.cpp"
         ],
         libraries=libraries,
-        include_dirs=["src/c++"],
-        library_dirs=[join(srcpath, "minpack"), join(srcpath, "port3")]
+        include_dirs=["src/c++", '/Users/rafferty/anaconda3/include'],
+        library_dirs=[join(srcpath, "minpack"), join(srcpath, "port3"), '/Users/rafferty/anaconda3/lib/', '/usr/local/gfortran/lib/']
     ))

[1054]

Thanks! Indeed it was also tricky for me to install it, and it would be great if this could be fixed in the next release! For the third fixing, I think using include_dirs = [..., sysconfig.get_config_var('INCLUDEDIR')] and library_dirs = [..., sysconfig.get_config_var('LIBDIR')] to replace the absolute paths would be better for future releases (this requires import sysconfig in the beginning).

[darafferty]

Ah -- good suggestion, thanks! Indeed, I'll try to have a look at it before the next release.

[keflavich]

Thanks! This worked for me, or at least, it got me through the installation step - I still have to try to use the code.

I'll leave the issue open as a reminder to add this into setup.py for the next release.

[keflavich]

Unfortunately, now I have problems at import time:

$ python -c "import bdsf"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/bdsf-1.9.0-py3.7-macosx-10.7-x86_64.egg/bdsf/__init__.py", line 21, in <module>
    from .collapse import Op_collapse
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/bdsf-1.9.0-py3.7-macosx-10.7-x86_64.egg/bdsf/collapse.py", line 11, in <module>
    from . import _cbdsm
ImportError: dlopen(/Users/adam/miniconda3/lib/python3.7/site-packages/bdsf-1.9.0-py3.7-macosx-10.7-x86_64.egg/bdsf/_cbdsm.cpython-37m-darwin.so, 2): Symbol not found: __ZN5boost6python7objects15function_objectERKNS1_11py_functionERKSt4pairIPKNS0_6detail7keywordES9_E
  Referenced from: /Users/adam/miniconda3/lib/python3.7/site-packages/bdsf-1.9.0-py3.7-macosx-10.7-x86_64.egg/bdsf/_cbdsm.cpython-37m-darwin.so
  Expected in: flat namespace
 in /Users/adam/miniconda3/lib/python3.7/site-packages/bdsf-1.9.0-py3.7-macosx-10.7-x86_64.egg/bdsf/_cbdsm.cpython-37m-darwin.so

which again looks like a boost-related problem.

[darafferty]

Indeed -- in my experience, this kind of error indicates some mismatch in the compilation of boost and that of _cbdsm -- there are (unfortunately) a number of issues, both open and closed, detailing similar problems. Maybe have a look at #11

[1054]

I did some tests and found a solution. Compiling with g++ leads to the exact error, while compiling with clang++ is fine. Anaconda seems using g++ in default. So the solution is three steps:

1. Uninstall bdsf and clean up build directory and bdsf/_*.so files before recompiling.
2. Edit "setup.py" and set the libraries = ['minpack', 'port3', 'gfortran.3', 'boost_numpy37', 'boost_python37']. Note that gfortran is gfortran.3 for anaconda.
3. Run in BASH shell: CXX=clang++ python setup.py install

Then import bdsf should work.

Actually, another small issue occurs after that, which is reporting numpy.core.umath failed to import. I think it is the anaconda libboost_numpy37.dylib issue and does not seem affecting any result.

[darafferty]

Wow, that's quite involved -- thanks for describing the process. I will add this info to the readme.

I also get the numpy.core.umath message, but indeed it seems to be harmless.

[keflavich]

This solution worked for me. I don't really understand it, as clang++ was the one in /usr/bin; I would've expected the anaconda c++ to work with anaconda libraries... but whatever works.

[emoravec]

For anyone else looking to install this on mac using anaconda, we were able to get this installed without (directly) using setup.py:

# Install Python 3.7 and bdsf dependencies
conda create -n pybdsf_37_env 'python<3.8' astropy scipy matplotlib ipython boost -y
# Enter environment
conda activate pybdsf_37_env
# Specify clang compiler and install bdsf itself via pip
CXX=clang++ pip install bdsf
# Test
python -c 'import bdsf'; echo $?

If CXX=clang++ is not specified, we get this error:

$ python -c 'import bdsf'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "~/repos/PyBDSF/bdsf/__init__.py", line 21, in <module>
    from .collapse import Op_collapse
  File "~/repos/PyBDSF/bdsf/collapse.py", line 11, in <module>
    from . import _cbdsm
ImportError: cannot import name '_cbdsm' from 'bdsf' (~/repos/PyBDSF/bdsf/__init__.py)

Though this doesn't appear to work on Python 3.8; we didn't test on any other versions. This is related to this issue: #170.