-
Notifications
You must be signed in to change notification settings - Fork 107
Installation FAQ (SCons)
This page covers questions about installing GalSim with SCons. For questions about installing with pip or setup.py, go here.
-
General questions, and basic SCons usage
- How do I "make clean" using SCons?
- How do I control the number of parallel jobs SCons uses?
- How do I force SCons to start over from scratch?
- How can I tell SCons where to look for certain header files or libraries?
- How do I get SCons to forget options I previously set?
- In Mac OSX, can I use fink/port to handle all my GalSim library dependencies?
- Can I use the LSST stack for my dependencies?
- Can I use Docker to containerize GalSim?
-
SCons fails to build
- What should I do about "ld: library not found for -lgomp" when building TMV on Mac OSX 10.9?
- Why can't SCons find TMV.h, even though I give it the correct directory as TMV_DIR?
- Why does SCons fail to link TMV, even though it finds the right tmv_link file?
- Why is SCons "Unable to get python include path python executable" on Ubuntu?
- Why is SCons "Unable to build a python loadable module using the python executable..."?
- Why is SCons "Unable to build a python loadable module that uses tmv" using Anaconda?
- Why is SCons unable to compile a file with #include "boost/python.hpp"?
- Why is SCons unable to run a program built with Boost.Python?
- Why is SCons unable to compile a file with numpy?
- Why does SCons report some error about a "bad value" and "recompile with -fPIC"?
- Why is SCons using the fftw library from a different directory than the one in FFTW_DIR?
- What should I do for a warning about "ld: warning: could not create compact unwind for..."?
- What should I do for an error "iomanip(64): error: expected an expression"?
- What should I do about "Undefined symbols for architecture x86_64:" errors?
- What should I do about "no newline at end of file" errors?
- What should I do about "friend declaration specifying a default argument must be a definition" errors?
-
Runtime and testing errors
- What should I do if "scons tests" crashes without finishing?
- What should I do if "scons tests" gives errors about wrong architecture?
- What should I do if "scons tests" fails with "terminate called after throwing an instance of 'galsim::integ::IntFailure'"?
- What should I do if "scons tests" gives errors about "libmkl_mc3.so" and "libmkl_def.so" having undefined symbols?
- What should I do if "scons tests" gives an error about "undefined symbol: omp_get_num_procs?
- Is it a problem if "scons tests" gets to the end of the dots, but then takes a really long time before finishing?
- What if I get a segmentation fault when I try to use GalSim?
-
Miscellaneous
scons -c will delete the local object files, libraries, and executables.
scons -c install will also delete the installed executables, libraries, and copied python files.
SCons has a -j flag just like make. So scons -j4 directs SCons to use 4 processes when compiling.
If you don't specify any -j flag, the GalSim SCons script will try to determine how many processors your system has and use that. This is normally the right value, but if you are running on a system with many other people running jobs as well, it might actually be slower to do this. So you might want to specify some smaller number of processes explicitly.
Also, when you do scons tests, we pass the number of jobs (either automatically determined or given by the -j flag) to nosetests, so it can run the tests in parallel as well. If you don't want this, you can do scons tests -j1, which will only use one process for running the unit tests.
Scons can detect when things change in the GalSim directory, but if you change things about your system (rebuild libraries, change symlinks, etc.) then it doesn't always notice, so it might not rebuild certain files that should be rebuilt. The best bet when you want to start SCons from a clean slate is to type
/bin/rm -rf .scon*
This makes SCons forget everything that it had learned about your system and redoes all the checks from scratch.
If scons is not automatically finding some of your include files for the various libraries, there are a few options.
The first choice solution is to use the scons flags like TMV_DIR to specify the location of the TMV library. This should be the root directory for TMV, so libtmv.so should be in TMV_DIR/lib and the header file TMV.h should be in TMV_DIR/include. Similarly, there are flags BOOST_DIR and FFTW_DIR for those libraries.
However, sometimes things get installed such that the above directory specification doesn't work. The general solution if SCons isn't finding a header file or a library with the above flags is to tell SCons about extra locations to look with the SCons parameters EXTRA_INCLUDE_PATH or EXTRA_LIB_PATH. e.g.
scons EXTRA_INCLUDE_PATH=/usr/include/fftw3
Another option that is sometimes useful if your sysadmin tends to use system variables like C_INCLUDE_PATH and LD_LIBRARY_PATH to specify where things are is to use the scons flag IMPORT_PATHS. This tells scons to import these paths as search paths for the compiler.
SCons will retain any flags that you specify on the command line until you change them to something else. This is usually helpful, since once you get the correct BOOST_DIR, TMV_DIR, etc. working, you don't want to have to specify them every time you recompile the code; however, it can occasionally be slightly unintuitive. If you want to remove something that you previously specified, say an EXTRA_FLAGS option that didn't work out, you should run with EXTRA_FLAGS="".
All the currently active (non-default) parameters are listed in the file gs_scons.conf. You can even edit them directly there if you want, which is sometimes easier. Especially if you just want to remove a parameter, it can be quicker to open the file, delete that line, and then type scons. Or if you need to edit a directory entry, it can be easier to edit it in the conf file, rather than retype the whole thing correctly.
Yes, except for TMV installation with port. Please refer to the file INSTALL.md for details.
After apparently successfully installing GalSim with fink, do you get a segmentation fault when doing import galsim?
(c.f. Issues #348)
This happened with OSX 10.8. We didn't figure out what caused this, but the workaround was to reinstall all the fink packages from scratch. You can do this with fink rebuild foo. Probably you need to do this for all the GalSim prerequisites: python27, python27-shlibs, numpy-py27, pyfits-py27, yaml-py27, and boost1.35.python27-shlibs. However, the user who reported issue #348 actually started over with a completely new fink installation, so we didn't confirm this solution.
In Issue #348, there was a problem with gcc47, which is part of GalSim's dependency tree on OSX 10.8. The failure message was:
Can't resolve dependency "xcode (>= 4.5)" for package "gcc47-4.7.2-1000" (no
matching packages/versions found)
Exiting with failure.
even though the user had installed XCode 4.5.2. Apparently (at least for 10.8), xcode to fink really means the XCode Command Line Tools. So, after installing a new version of XCode, you also need to open XCode, go to Preferences, Downloads tab, and update the "Command Line Tools" there. This fixed the problem in that case.
Yes. GalSim is even available now as part of the LSST stack as an optional installation.
See a write-up from Chris Walter about how to do this, including some information for dealing with issues related to Anaconda.
Yes. Karen Ng has very kindly made available her Docker file for building GalSim in a Docker container. See her GitHub repo for more information.
(c.f. Issues #493)
In short, in OSX 10.9, Apple moved to clang++ as the default compiler, but calls it g++. So, TMV tries to use OpenMP which is supported by g++ but not clang++. To address this for now, explicitly tell scons that you are using clang++ when compiling TMV and GalSim, i.e., use the CXX=clang++ option. The next release of TMV will probably fix this problem.
First, TMV_DIR should not be the directory where "TMV.h" is located. Rather, the file "TMV.h" should be in TMV_DIR/include. In other words, TMV_DIR should match the PREFIX given during TMV installation. And the TMV library files should be in TMV_DIR/lib.
If all of that seems to be correct, it is possible that SCons is actually finding the right file, but there is a problem when it tries to compile a simple program using it. Look in your config.log file to see what might be going on when SCons tries to compile with TMV.
Are there errors about "error: explicit instantiation of ‘std::basic_ostream<_CharT, _Traits>& std::operator<<(...)"?
This error can show up after upgrading from Mac OSX 10.6 or earlier to 10.7 or later, when using a custom python build (e.g., the EPD python) and trying to compile with g++ (a new version after re-installing Xcode for the new OS). It is a more general problem involving the C++ header files changing in incompatible ways, and g++ erroneously using the old headers. (Only within SCons though -- the commands will work from the command line.) In the GalSim installation, this error first shows up when trying to compile with TMV, so that's why GalSim reports it as an error with TMV.
The reason for this failure is that Scons is based on python, and if your python was built in OSX 10.5 or 10.6, it will include an extra environment variable MAXOSX_DEPLOYMENT_TARGET=10.5. This variable is not set if you are compiling using g++ from the command-line. When that variable is set, the system tries to compile using the old header files, which leads to compilation failures when using a new g++ that was built with the new OS.
There are a few options here:
- Run scons as
scons EXTRA_FLAGS=-mmacosx-version-min=10.7. - Use clang:
scons CXX=c++, which reportedly does not have this problem. - Use a version of SCons that uses the system python, which should have been upgraded when the OS was upgraded.
Note: You probably need to rebuild TMV as well using the same fix.
(cf. Issue #517 )
XCode version 5.1 included an upgrade to clang that doesn't like some of the v0.71 TMV header files. In GalSim, this shows up as failing to find the TMV header, since using the header fails to compile. The solution is to update your TMV installation to v0.72.
(c.f. Issues #203) Some of the lines in the config.log file reported in issue #203 were:
ld: warning: in /sw/lib/libtmv_symband.dylib, file was built for i386 which is not the architecture
being linked (x86_64)
ld: warning: in /sw/lib/libfftw3.dylib, file was built for i386 which is not the architecture being
linked (x86_64)
ld: warning: in /sw/lib/libtmv.dylib, file was built for i386 which is not the architecture being
linked (x86_64)
The problem here is that some systems intermix 32 bit and 64 bit libraries. In this case, it was a Mac running Snow Leopard, but there may be other systems with similar issues. The TMV libraries installed by fink were installed as 32 bit libraries. But for GalSim, g++ was building 64 bit binaries. Since they didn't match, g++ ignores the discrepant libraries and then gets linking errors.
In this case, the recommendation was to switch fink to 64-bit. This is usually what you want, even though the default installation for Snow Leopard is 32-bit. You can do this either as two separate fink setups (typically installed in /sw and /sw64), or just overwriting the current 32-bit version with a new 64-bit installation. See here for more information.
If you are seeing this kind of error and you're not using fink, you just need to find some way to install the library with the right bit-ness. Usually, this would just involve re-installing the problematic library and make sure you use the same compiler that you are using for GalSim. This should usually do the trick.
(c.f. Issues #314) The TMV library has been built linking to the Intel Math Kernel Library (MKL) optimised implementations of the Basic Linear Algebra Subprograms (BLAS). However, when linking to TMV built in this way gives errors such as:
Output of the command .sconf_temp/conftest_6 is:
.sconf_temp/conftest_6: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
Output of the command .sconf_temp/conftest_7 is:
.sconf_temp/conftest_7: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
Output of the command .sconf_temp/conftest_8 is:
.sconf_temp/conftest_8: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
Output of the command .sconf_temp/conftest_9 is:
.sconf_temp/conftest_9: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
As discussed in Issue #314, the current best workaround is to install TMV using the (non-default!) scons WITH_BLAS=False option. TMV will then use its own implementation of the BLAS.
This will not cause a large slowdown in GalSim, as the use of the TMV linear algebra suite is fairly minimal. We will update this FAQ if a better solution is found to this issue of using TMV+MKL+GalSim.
Another option may be to use icpc as your compiler for both TMV and GalSim. This usually works fine, but for the particular system in Issue #314, the user ran into a bug with icpc that prevented GalSim from compiling, so that turned out not to work either.
(cf. Issue #721 and #725 )
Starting with El Capitan, Apple decided to disallow certain "dangerous" environment variables from being used by system calls. They call this new "feature" System Integrity Protection (SIP) if you want to look up more information about it. Unfortunately, calling scons counts as such a system call and SIP strips out many of the environment variables so SCons doesn't have access to them.
Among other variables, this includes the environment variable DYLD_LIBRARY_PATH, which some setups depend on to figure out the locations of libraries at run time. Thus, when SCons tries to test whether things are working properly, the system call doesn't find the libraries that it needs and it fails.
This is particularly a problem with Anaconda installations, including the LSST stack's conda installation of GalSim, which needs DYLD_LIBRARY_PATH to be set correctly. So if you are trying to install a developer branch of GalSim using Anaconda python, this may affect you.
The solution we came up with is to tell SCons directly what your DYLD_LIBRARY_PATH is with a scons option, rather than just let it find it through the normal os.environ, since it will have been stripped out by the SIP stuff. Run scons as follows:
scons DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
or if you are using the LSST stack's installation, you should also have a variable called LSST_LIBRARY_PATH which you can use instead:
scons DYLD_LIBRARY_PATH=$LSST_LIBRARY_PATH
Please see this stackoverflow question and answer.
SCons outputs the contents of the tmv-link file that it finds after a line that says
Using TMV_LINK file:...
If this includes something like -Lshare/~/lib then this means that when you were installing TMV, SCons failed to expand the ~ into your home directory as it usually does. This manifests at runtime when python tries to load the TMV library, and it gives an error like:
ImportError: ~/lib/libtmv_symband.so.0: cannot open shared object file: No such file or directory
(You should see this error in your config.log file somewhere around conftest_9.)
The solution is to recompile TMV, specifying the full directory name of your home directory:
scons PREFIX=/my/home/directory/
If you are using gcc version 4.2 on linux (or possibly earlier, but I think just 4.2), then there was a bug in their libgomp.so file that made it not dynamically loadable. This manifests as an error message:
ImportError: libgomp.so.1: shared object cannot be dlopen()ed
for conftest_8 (or thereabouts) in the file config.log. This bug was apparently fixed in gcc version 4.3, so one solution would be to upgrade to that version (or later). But the gomp library is only used by TMV. So an easier solution may be to rebuild TMV with scons WITH_OPENMP=false, which should fix the problem.
It seems that there is a bug in Anaconda Python where it doesn't install its libraries in a normal way, so GalSim can have trouble linking to the right Python libraries. This was reported in this StackOverflow question, which in turn points to this question, which has a hacky solution that was claimed to work.
See also a write-up from Chris Walter about installing GalSim with the LSST stack that deals with this problem.
And if all that fails, you may need to just forego the Anaconda boost and install boost from source.
See this earlier FAQ entry, which may be relevant here.
/numeric_op.hpp:290:31: error: missing binary operator before token "("
If so, this is due to older boost libraries being incompatible with newer g++ compilers. e.g. Using the fink boost1.35.python27 with g++ 4.7 is known to cause this problem. Here are the tickets about this problem for gcc and for boost. It looks like gcc won this argument, so boost had to change.
The solution is to either use an older g++ (<= 4.3) or a newer boost (>= 1.37). If you are using fink, and you want to stick with g++ 4.7 for some reason, then you'll need to use fink's boost1.46.1.cmake. However, this will use the system python, not fink's python. So you'll need to make sure numpy and pyfits (at least) are installed for that version of python. These generally need to be done by hand. (Although installation is very easy for them, so it's not too onerous.)
Anaconda ships a bunch of extra libraries in its lib directory. This includes libraries that really belong to the compiler like libgomp.so, libstdc++.so, libiomp.so, etc. If the compiler you used to compile TMV is more recent than what Anaconda used for its own libraries, it might give errors (in the gs_error.txt file) like
ImportError: /home/esheldon/anaconda3/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found
(required by .sconf_temp/conftest_31_mod/check_tmv.so)
Erin Sheldon reported this in Issue #842, so you may want to read through the discussion there for the various things that were tried with varying levels of success. Some of those might work for your system.
Erin's eventual solution was to switch to miniconda rather than the full Anaconda, since that doesn't include the problematic libstdc++ files:
- Install tmv in the standard way.
- Install miniconda http://conda.pydata.org/miniconda.html
- Install basic packages needed for galsim
conda install future numpy astropy pyyaml - Install galsim in the usual way, specifying
TMV_DIR
Another solution known to work is to downgrade the gcc version to 4.x so that the library that anaconda ships is compatible with the one being used by gcc.
Read the answer to this question, since that answer is probably relevant here. In particular, you probably want to install boost with
./bootstrap.sh
./b2 toolset=clang cxxflags="-stdlib=libc++" linkflags="-stdlib=libc++" install
Apparently, there is no 100% reliable way to determine which python version a boost library uses. However, you may be able to figure it out by typing one of the following:
ldd <BOOST_LIBDIR>/libboost_python<POSSIBLE_SUFFIX>.so (on Linux)
otool -L<BOOST_LIBDIR>/libboost_python<POSSIBLE_SUFFIX>.dylib (on OSX)
If the Python library listed is the one you will be using, all is well. If not, Boost can be forced to use a different version by specifying the following options to the ./bootstrap.sh installation script (defaults in [] brackets):
--with-python=<PYTHON> specify the Python executable [python]
--with-python-root=<DIR> specify the root of the Python installation [automatically detected, but some users have found they have to force it to use a specific one because it detected the wrong one]
Even this sometimes doesn't work though. If you have already installed boost once without the above flag(s), then you need to use the -a flag when you run b2 to tell it to recompile the libraries rather than just use the existing libraries.
If you
-
Installed Boost.Python into a non-standard location, e.g. somewhere in your home directory. This may be done to avoid requiring root/admin access to write into the normal locations such as
/usr/local. And... -
Are getting build error messages something like (Linux):
.sconf_temp/conftest_10: error while loading shared libraries: libboost_python.so.1.49.0: cannot open
shared object file: No such file or directory
Checking if we can build against Boost.Python... no
Cannot run program built with Boost.Python.
or (Mac OSX):
dyld: Library not loaded: libboost_python.dylib
Referenced from:
/Users/Rick/Desktop/GalSim-developers-GalSim-651eca4/.sconf_temp/conftest_10
Reason: image not found
sh: line 1: 41400 Trace/BPT trap: 5 .sconf_temp/conftest_10 >
".sconf_temp/conftest_10.out"
Checking if we can build against Boost.Python... no
Cannot run program built with Boost.Python.
...then you may simply be suffering from a library linking issue that can be solved (as a temporary, stopgap measure) by adding the Boost.Python library location to the environment variable LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (Mac OSX). See the following link for some details in Linux: http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html (Section 3.5); and the following link for some details in Mac OSX: https://developer.apple.com/library/mac/#documentation/Darwin/Reference/Manpages/man1/dyld.1.html.
Look at the end of the config.log file that SCons creates. Do you see things about using a deprecated numpy API?
(c.f. Issues #298 and #373)
This is caused by using NumPy version newer than 1.7, currently the latest one we support. They changed the API pretty significantly with this version and deprecated much of the old API. We fixed our code to work with the new API as of version 1.7. But the development branch (to become version 1.8) doesn't work with our code as of this writing. So if you have a development branch version of numpy, you should downgrade to version 1.7.
If you have this problem, and the numpy version you are using is considered a stable release (i.e. it is available on SourceForge and there is no "rc" or "b" after the version number), then please file an Issue for us to look into this again.
Python loads all of its libraries at run time, which means that all the libraries need to be shared libraries, rather than the static libraries that you might be used to. Usually, this isn't a problem as most distributions include shared libraries by default. However, occasionally one might come only as a static library.
It's still possible that this is ok if it was compiled with "position independent code" (PIC). It just means that we need to explicitly include the objects in the static library in with our GalSim library so that it can be loaded at runtime by python. So the builder tries to do that.
However, if the static library wasn't compiled with the "-fPIC" flag, then this will fail with a message something like the following:
/usr/bin/ld: /usr/local/lib/libfftw3.a(mapflags.o): relocation R_X86_64_32 against `.rodata' can not be
used when making a shared object; recompile with -fPIC
/usr/local/lib/libfftw3.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
scons: *** [lib/libgalsim.so.0] Error 1
If you get a message similar to this, then it means that you need to re-install the library in question. In the above example it is the fftw3 library. You need to re-install the library with whatever options are necessary to get it to build a shared library rather than a static library. However, before you do, you might check your system to see if there is a shared version of the library in some other directory that you could specify (with FFTW_DIR for example).
The way SCons looks for libraries is the same way compilers normally look for libraries. We specify the library paths with -L then the compiler looks for each library in those directories. The order that the *_DIR directories will be searched are BOOST_DIR, then TMV_DIR, then FFTW_DIR. So if you have one fftw library in BOOST_DIR, say, and another one in a different directory, specified by FFTW_DIR, then the one in BOOST_DIR will be found first. If that library is incompatible with the compiler or something, then SCons will fail and say it can't find a good fftw library.
The logic of the above order is that you are most likely to want a special version of boost, which might need a special directory, then TMV, then finally FFTW, for which it is usually the case that whatever version is installed in a standard directory is probably fine. But of course that's not always the case.
The possible solutions are (1) remove the irrelevant version or re-install it in a directory that SCons does not see, or (2) install versions of the other libraries (boost or tmv) that are in that directory other places, so you don't need to specify that directory for anything.
If the library is in a system location like /usr/lib64, and you aren't a sysadmin on the system in question, and the sysadmin doesn't want to install a new version, then you might be out of luck as far as the above options go. You may need to just use that version of the fftw library. If the conflict is due to a compiler mismatch, you may then be forced to use the same compiler for GalSim. If the conflict is because that fftw library was not compiled with -fPIC, I'm afraid you will just need to be more persistent in pestering your sysadmin.
This seems to be a Mac problem with the linker on some Macs with gcc v4.7. The solution is to use:
scons EXTRA_FLAGS='-Wl,-no_compact_unwind'
It is advisable to remove these flags if you no longer wish to compile with gcc v4.7, as it is known to cause issues with, e.g., gcc v4.0.
(c.f. Issues #314) The error may look like this:
icpc -o src/.obj/Laguerre.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/Laguerre.cpp
/usr/local/include/c++/4.5.0/iomanip(64): error: expected an expression
{ return { __mask }; }
^
/usr/local/include/c++/4.5.0/iomanip(94): error: expected an expression
{ return { __mask }; }
^
[...]
This can happen for some systems using the icpc compiler. Apparently it is a known bug in icpc when the gcc library has some C++11 features in it, and icpc is slightly older (<12.0). In the discussion about the bug, they claim that upgrading to icpc version 12 and compiling with the flag -std=c++0x will fix the problem.
You can add that flag to GalSim's compilation with:
scons EXTRA_FLAGS=-std=c++0x
First, you should read about the problem with clang++ masquerading as g++ here. However, there may still be problems even after explicitly using CXX=clang++. If so, keep reading...
(c.f. Issue #493) This seems to happen when two libraries (i.e. GalSim and something else like boost) are trying to use different c libraries. With 10.9, Mac decided to have two different incompatible C standard libraries. If a library is installed with one of them, and one tries to link it into a program or library that uses the other one, you get this not-terribly-helpful error message.
Here are some sites that discuss this issue in more detail:
The last one recommends installing boost via:
./bootstrap.sh
./b2 toolset=clang cxxflags="-stdlib=libc++" linkflags="-stdlib=libc++" install
which seems to work. Eventually, we will try to automatically detect this and either give a helpful error message or add some flags to the GalSim installation to make this work seamlessly. But we have not done so yet.
Some boost installations have an error in their auto-built file "tuple.hpp" and related files, where they don't get the newline at the end. For header files, the newline is required, otherwise things can join up improperly in the .cpp file where they are included, so g++ spits out errors like:
In file included from /home/mjarvis/include/boost/fusion/tuple/tuple.hpp:22,
from /home/mjarvis/include/boost/fusion/tuple.hpp:10,
from /home/mjarvis/include/boost/fusion/include/tuple.hpp:10,
from /home/mjarvis/include/boost/math/tools/tuple.hpp:89,
from /home/mjarvis/include/boost/math/special_functions/detail/igamma_inverse.hpp:13,
from /home/mjarvis/include/boost/math/special_functions/gamma.hpp:1651,
from src/SBSersic.cpp:31:
/home/mjarvis/include/boost/fusion/tuple/detail/preprocessed/tuple.hpp:21:7: error: no newline at end of file
In file included from /home/mjarvis/include/boost/fusion/tuple/make_tuple.hpp:27,
from /home/mjarvis/include/boost/fusion/tuple.hpp:11,
from /home/mjarvis/include/boost/fusion/include/tuple.hpp:10,
from /home/mjarvis/include/boost/math/tools/tuple.hpp:89,
from /home/mjarvis/include/boost/math/special_functions/detail/igamma_inverse.hpp:13,
from /home/mjarvis/include/boost/math/special_functions/gamma.hpp:1651,
from src/SBSersic.cpp:31:
/home/mjarvis/include/boost/fusion/tuple/detail/preprocessed/make_tuple.hpp:21:7: error: no newline at end of file
... etc.
The solution is either to install a more recent version of boost (they apparently fixed the problem in 1.50.0) or to manually add a newline at the end of each of the files mentioned in the error message. Here is a quick bash command that will do this for you:
for f in [INCLUDE_PATH]/boost/fusion/tuple/detail/preprocessed/*.hpp; do echo >> $f; done
Or if you prefer python:
import glob
for file in glob.glob('[INCLUDE_PATH]/boost/fusion/tuple/detail/preprocessed/*.hpp'):
f = open(file,'a')
print >> f, '\n'
f.close()
What should I do about "friend declaration specifying a default argument must be a definition" errors?
(c.f. Issue #517)
With the latest version of XCode, you may see errors that look like:
include/tmv/TMV_Permutation.h:471:28: error: friend declaration specifying a default argument must be a definition
friend inline void QRP_Decompose(
^
include/tmv/TMV_Permutation.h:476:28: error: friend declaration specifying a default argument must be a definition
friend inline void QRP_Decompose(
^
include/tmv/TMV_Permutation.h:852:17: error: friend declaration specifying a default argument must be the only declaration
inline void QRP_Decompose(
^
include/tmv/TMV_Permutation.h:471:28: note: previous declaration is here
friend inline void QRP_Decompose(
^
This started happening when OSX upgraded to clang version 5.1. Clang used to allow this particular construction, but apparently it's some edge case in the C++ standard that is technically illegal. Rather than just give a warning about it, they decided to go straight to making it a compiler error.
Anyway, it was fixed in TMV version 0.72, so the solution is to install that.
===
The first thing to do is to run nosetests yourself with more verbose output to narrow down where the problems is. So:
scons install
cd tests
nosetests -v
Do you get something with "assertion !pthread_create(&thr->thrH, &attr, rout, arg) failed, line 55 of file ..."?
This seems to indicate a problem with your ATLAS installation. Two possibly useful links to help you fix the problem are:
http://sourceforge.net/p/math-atlas/support-requests/873/
http://math-atlas.sourceforge.net/atlas_install/node21.html
If you don't want to go through the effort of figuring this out, then another solution is to rebuild TMV to use either a different BLAS if there is another one on your system (see the TMV manual for options for doing this), or no BLAS library at all (with the option WITH_BLAS = false).
For example, you might get errors that look like:
ERROR: Failure: ImportError (dlopen(/Users/Mike/GalSim/galsim/_galsim.so, 2): no suitable image found.
Did find:
/Users/Mike/GalSim/galsim/_galsim.so: mach-o, but wrong architecture)
This happens when you have both 32-bit and 64-bit libraries on your system, similar to an issue above about incompatible architecture, except now it's happening at run-time from a mismatch between the python bit-ness and the GalSim library you just compiled. You can change from 32-bit to 64-bit by typing (for bash shells):
export VERSIONER_PYTHON_PREFER_64_BIT=no; export VERSIONER_PYTHON_PREFER_32_BIT=yes'
To go back to 32-bit, you would type
export VERSIONER_PYTHON_PREFER_32_BIT=no; export VERSIONER_PYTHON_PREFER_64_BIT=yes'
On my system with both 32 and 64 bit stuff, I have these aliased to py32 and py64 respectively to make it easy to switch.
The other option is to make sure your compiler is building libraries that use the bit-ness that matches your default python. You can do this by adding a flag to the compiler. e.g.
scons EXTRA_FLAGS='-m64'
to force it to build 64-bit libraries. Note: all the other libraries you will be linking to also have to match.
What should I do if "scons tests" fails with "terminate called after throwing an instance of 'galsim::integ::IntFailure'"
(c.f. Issue #290)
This was found to happen when using MacPorts' g++ 4.7 ("mp-gcc47") to install GalSim when some of the other MacPorts modules were compiled and installed with the system g++ 4.0 ("gcc40"). There were two versions of libgcc_s listed when doing otool -L lib/libgalsim.dylib, which is suspected to be the problem.
The solution in that case was to switch to using the system g++ (v4.0 or v4.2) for installing GalSim. (This problem seems to manifest itself only on Mac OS X 10.5.8, but not on 10.7.4.)
What should I do if "scons tests" gives errors about "libmkl_mc3.so" and "libmkl_def.so" having undefined symbols?
(c.f. Issue #261)
This issue should be fixed now, but it can still show up if you change your SCons parameters after a successful build, since SCons may not realize that it needs to redo certain tests. So the first thing to try is /bin/rm .scon* and then rebuild.
If this doesn't work, the basic problem is that certain extra libraries need to be linked for modules that use MKL that aren't necessary for executables. So the TMV installation didn't include them in the list of required libraries. If SCons isn't figuring this out correctly, you can try adding them yourself with one of the following (or perhaps a different library -- look in the MKL lib directory on your system to see what might be a good library name to try for your version of MKL).
scons EXTRA_LIBS='mkl_rt'
scons EXTRA_LIBS='mkl_mc'
scons EXTRA_LIBS='mkl_mc3'
etc.
This happens to me on one system where I regularly test GalSim with both icpc and g++. When I switch, I also have to do /bin/rm -rf .scon* to make SCons forget about the MKL runtime libraries that icpc needs. Otherwise I get a runtime error:
../usr/bin/python2.6: symbol lookup error: /opt/intel/composerxe-2011.4.191/mkl/lib/intel64/libmkl_i
ntel_thread.so: undefined symbol: omp_get_num_procs
Nosetests returned error code 127
Is it a problem if "scons tests" gets to the end of the dots, but then takes a really long time before finishing?
Probably. This may be a problem with the numpy module having adverse interactions with pyfits. Here is a test you can do to see if this is a problem on your system:
cd tests
nosetests --processes=1 -s test_wcs.py
You should see a number of progress messages. If it stops at
Start image tests for WCS GSFitsWCS_TAN
and then just hangs, then this is a problem. (When you run scons tests there is a timeout limit, so it doesn't hang forever. I thought it was supposed to raise a TimedOutException when a thread times out, but on my system it didn't do that for some reason.)
This happened to me on a MacOS 10.9 machine using both the system python and fink python. I narrowed the problem down to a call to numpy.dot. When I started stripping away code to make a self-contained bug report, I discovered that the bug went away when I stopped importing pyfits.
I don't exactly know what is going on, but I suspect it has something to do with Apple's POS BLAS library. It's been buggy forever, and they still haven't fixed it -- hanging when in multithreaded mode being one of its bugs. I think numpy normally uses an atlas library for its BLAS stuff, but pyfits might use Apple's BLAS library, so if that gets imported, the BLAS commands get dispatched to Apple's BLAS library instead, which then hangs forever. This is just a guess really, but it seems plausible given the behavior I've been seeing.
The solution for me was to install astropy and scipy. (The latter is mostly because astropy.wcs requires scipy, so we gain that functionality as well.) Once I did that, GalSim finds astropy.io.fits and prefers that to pyfits, so we don't end up importing pyfits, and the hang up doesn't happen anymore.
Seg faults usually mean that the header files used for things to compile GalSim don't correspond to the libraries specified when linking (or possibly to the libraries loaded at runtime by python). This can happen sometimes if you have multiple copies of some of the dependencies sitting around and the one in your LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH) is not the same as what you told GalSim to use (or what it found on its own) when compiling.
If you are using Anaconda on a Mac, it's likely that their boost installation is the problem. Possibly these instructions will be helpful to workaround this issue.
If not (or if the above suggestion didn't work), the way to track down this kind of problem is usually to carefully check the library dependencies of the compiled libraries. On a Mac, you check with otool -L. On linux with ldd.
Start with that command on both lib/libgalsim.so (or .dylib on the Mac) and also galsim/_galsim.so. You're looking for a case where either the directory for the library is not explicit, so it will just take the first thing in the path or where the explicit directory is not the one you meant for GalSim to use when compiling. Then it may be the case that the headers don't properly match the loaded library.
The most common culprit for this kind of thing is the boost library, so pay special attention to that. I promise, someday, we'll get rid of boost as a dependency, and in the meanwhile we apologize that it tends to give so many people trouble.
===
On some Mac systems, scons install will output some warnings that look something like:
ld: warning: can't add line info to anonymous symbol cstring=p == 0 || p != px from src/.obj/SBSersic.os
ld: warning: can't add line info to anonymous symbol cstring=px != 0 from src/.obj/SBLaguerre.os
ld: warning: can't add line info to anonymous symbol cstring=Interpolant2d::shoot() not implemented for this kernel from src/.obj/SBInterpolatedImage.os
ld: warning: can't add line info to anonymous symbol galsim::SBDeconvolve::SBDeconvolveImpl::~SBDeconvolveImpl() [clone .lsda] from src/.obj/SBDeconvolve.os
ld: warning: can't add line info to anonymous symbol tmv::DiagMatrixView<std::complex<double>, 0>::cdiag() const [clone .lsda] from src/.obj/Table.os
ld: warning: can't add line info to anonymous symbol cstring=HSM Error: from src/hsm/.obj/PSFCorr.os
You can ignore these. Apparently, it is a bug in the XCode 3.x linker that usually shows up for i386 systems, but not x86_64 systems. The warning was removed in XCode 4.0. These warnings should not prevent GalSim from building correctly.
In Mac OSX, what can I do if I'm having trouble getting Boost.Python to work with Enthought Canopy Python?
Enthought "Canopy" seems to be the future of the Enthought Python Distribution, but it has thrown up some issues when building the Boost.Python library. @dkirkby has very kindly documented here the steps necessary to build Boost.Python with Canopy 1.3.
The reason that building Boost.Python fails is that b2 cannot find the standard python distribution headers (pyconfig.h, etc.) under <base>/include/python2.7/. The simplest fix is to just add a soft-link in your personal Canopy tree. In @dkirkby's case for canopy-1.3.0.1715, this was:
cd ~/canopy/include
ln -s /Applications/Canopy.app/appdata/canopy-1.3.0.1715.macosx-x86_64/Canopy.app/Contents/include/python2.7
Note: when using a different Canopy version number, the line above should of course be modified to point to the installed version you wish to link. The above is just an example case for version 1.3.0.1715.
Also, your Canopy directory might be found instead at a directory that looks something like:
~/Library/Enthought/Canopy_64bit/System/include
If you don't want to modify your personal Canopy tree, you could instead add the necessary include path to the boost build config. This involves adding the following line to <BOOST>/tools/build/v2/user-config.jam (note that spaces are important here):
using darwin : : : <compileflags>-I/Applications/Canopy.app/appdata/canopy-<version_number>.macosx-x86_64/Canopy.app/Contents/include/python2.7/ ;
In @dkirkby's case the Canopy <version_number> was again 1.3.0.1715, but this should be modified to the value describing the Canopy version you wish to link to.
With either approach, you can then build with:
./bootstrap.sh --with-libraries=python,math
./b2 -a link=shared
sudo ./b2 link=shared install
Note that the resulting libboost_python.dylib library uses the @rpath mechanism to find the Enthought distribution's link library at runtime (which, for some reason, wasn't the case for @dkikby if Boost.Python built against a python distribution under /System/Library/Frameworks/Python.framework/):
otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python.dylib:
libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/Python (compatibility version 2.7.0, current version 2.7.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)