TMV linking failure not present in v0.2 but present now

[barnabytprowe]

As in #312, I think must be having issues with new SCons options too, I believe. On my 64bit linux system, if I checkout v0.2 and run scons I link to TMV fine:

lucius% scons
scons: Reading SConscript files ...
SCons is version 2.0.1 using python version 2.7.2
Python is from /usr/local/EPD/epd-7.1.2/include/python2.7
Using the following (non-default) scons options:
   CXX = /usr/bin/g++
   TMV_DIR = /home/browe/local/64
These can be edited directly in the file gs_scons.conf.
Type scons -h for a full list of available options.
Using python =  /usr/bin/env python
Using default PYPREFIX =  /usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages
Using compiler: /usr/bin/g++
compiler version: 4.4.6
Determined that a good number of jobs = 24
Checking for C++ library fftw3... yes
Checking for C++ header file boost/shared_ptr.hpp... yes
Checking for C++ header file TMV.h... yes
Using TMV_LINK file: /home/browe/local/64/share/tmv/tmv-link
     -L/home/browe/local/64/lib -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -Wl,rpath=/home/browe/local/64/lib -fopenmp
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage... yes

But then immediately checking out the current master, I get:

lucius% rm -rf .scon*
lucius% scons
scons: Reading SConscript files ...
SCons is version 2.0.1 using python version 2.7.2
Python is from /usr/local/EPD/epd-7.1.2/include/python2.7
Using the following (non-default) scons options:
   CXX = /usr/bin/g++
   TMV_DIR = /home/browe/local/64
These can be edited directly in the file gs_scons.conf.
Type scons -h for a full list of available options.
Using python =  /usr/bin/env python
Using default PYPREFIX =  /usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages
Using compiler: /usr/bin/g++
compiler version: 4.4.6
Determined that a good number of jobs = 24
Checking for C++ header file fftw3.h... yes
Checking for correct FFTW linkage... yes
Checking for C++ header file boost/shared_ptr.hpp... yes
Checking for C++ header file TMV.h... yes
Using TMV_LINK file: /home/browe/local/64/share/tmv/tmv-link
     -L/home/browe/local/64/lib -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -Wl,rpath=/home/browe/local/64/lib -fopenmp
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage... 
Error: TMV file failed to link correctly
Check that the correct location is specified for TMV_DIR

Please fix the above error(s) and re-run scons

The TMV link files are the same. The relevant lines from the (failing) config.log are:

scons: Configure: Checking for correct TMV linkage... 
.sconf_temp/conftest_5.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_5.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/home
/browe/local/64/include .sconf_temp/conftest_5.cpp
.sconf_temp/conftest_6.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_6.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/home
/browe/local/64/include .sconf_temp/conftest_6.cpp
/usr/bin/g++ -o .sconf_temp/conftest_6 -fopenmp .sconf_temp/conftest_6.o -Llib -L/home/browe/local/6
4/lib -ltmv_symband -lfftw3 -lpthread -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread
.sconf_temp/conftest_6 > .sconf_temp/conftest_6.out
.sconf_temp/conftest_6: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shar
ed object file: No such file or directory
.sconf_temp/conftest_7.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_7.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/home
/browe/local/64/include .sconf_temp/conftest_7.cpp
/usr/bin/g++ -o .sconf_temp/conftest_7 -Wl,-rpath=#lib -fopenmp .sconf_temp/conftest_7.o -Llib -L/ho
me/browe/local/64/lib -ltmv_symband -lfftw3 -lpthread -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequen
tial -lpthread
.sconf_temp/conftest_7 > .sconf_temp/conftest_7.out
.sconf_temp/conftest_7: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shar
ed object file: No such file or directory
.sconf_temp/conftest_8.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_8.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/home
/browe/local/64/include .sconf_temp/conftest_8.cpp
/usr/bin/g++ -o .sconf_temp/conftest_8 -Wl,-rpath=/home/browe/local/64/lib -fopenmp .sconf_temp/conf
test_8.o -Llib -L/home/browe/local/64/lib -ltmv_symband -lfftw3 -lpthread -ltmv -lmkl_intel_lp64 -lm
kl_core -lmkl_sequential -lpthread
.sconf_temp/conftest_8 > .sconf_temp/conftest_8.out
.sconf_temp/conftest_8: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shar
ed object file: No such file or directory
.sconf_temp/conftest_9.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_9.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/home
/browe/local/64/include .sconf_temp/conftest_9.cpp
/usr/bin/g++ -o .sconf_temp/conftest_9 -fopenmp .sconf_temp/conftest_9.o -Llib -L/home/browe/local/6
4/lib -ltmv_symband -lfftw3 -lpthread -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread
.sconf_temp/conftest_9 > .sconf_temp/conftest_9.out
.sconf_temp/conftest_9: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shar
ed object file: No such file or directory
.sconf_temp/conftest_10.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_10.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/hom
e/browe/local/64/include .sconf_temp/conftest_10.cpp
/usr/bin/g++ -o .sconf_temp/conftest_10 -Wl,-rpath=#lib -Wl,-rpath=/home/browe/local/64/lib -fopenmp
 .sconf_temp/conftest_10.o -Llib -L/home/browe/local/64/lib -ltmv_symband -lfftw3 -lpthread -ltmv -l
mkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread
.sconf_temp/conftest_10 > .sconf_temp/conftest_10.out
.sconf_temp/conftest_10: error while loading shared libraries: libmkl_intel_lp64.so: cannot open sha
red object file: No such file or directory
.sconf_temp/conftest_11.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_11.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/hom
e/browe/local/64/include .sconf_temp/conftest_11.cpp
/usr/bin/g++ -o .sconf_temp/conftest_11 -fopenmp .sconf_temp/conftest_11.o -Llib -L/home/browe/local
/64/lib -ltmv_symband -lirc -limf -lfftw3 -lpthread -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequenti
al -lpthread
.sconf_temp/conftest_11 > .sconf_temp/conftest_11.out
.sconf_temp/conftest_11: error while loading shared libraries: libmkl_intel_lp64.so: cannot open sha
red object file: No such file or directory
.sconf_temp/conftest_12.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_12.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/hom
e/browe/local/64/include .sconf_temp/conftest_12.cpp
/usr/bin/g++ -o .sconf_temp/conftest_12 -Wl,-rpath=#lib -fopenmp .sconf_temp/conftest_12.o -Llib -L/
home/browe/local/64/lib -ltmv_symband -lirc -limf -lfftw3 -lpthread -ltmv -lmkl_intel_lp64 -lmkl_cor
e -lmkl_sequential -lpthread
.sconf_temp/conftest_12 > .sconf_temp/conftest_12.out
.sconf_temp/conftest_12: error while loading shared libraries: libmkl_intel_lp64.so: cannot open sha
red object file: No such file or directory
.sconf_temp/conftest_13.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_13.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/hom
e/browe/local/64/include .sconf_temp/conftest_13.cpp
/usr/bin/g++ -o .sconf_temp/conftest_13 -Wl,-rpath=/home/browe/local/64/lib -fopenmp .sconf_temp/con
ftest_13.o -Llib -L/home/browe/local/64/lib -ltmv_symband -lirc -limf -lfftw3 -lpthread -ltmv -lmkl_
intel_lp64 -lmkl_core -lmkl_sequential -lpthread
.sconf_temp/conftest_13 > .sconf_temp/conftest_13.out
.sconf_temp/conftest_13: error while loading shared libraries: libmkl_intel_lp64.so: cannot open sha
red object file: No such file or directory
.sconf_temp/conftest_14.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_14.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/hom
e/browe/local/64/include .sconf_temp/conftest_14.cpp
/usr/bin/g++ -o .sconf_temp/conftest_14 -fopenmp .sconf_temp/conftest_14.o -Llib -L/home/browe/local
/64/lib -ltmv_symband -lirc -limf -lfftw3 -lpthread -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequenti
al -lpthread
.sconf_temp/conftest_14 > .sconf_temp/conftest_14.out
.sconf_temp/conftest_14: error while loading shared libraries: libmkl_intel_lp64.so: cannot open sha
red object file: No such file or directory
.sconf_temp/conftest_15.cpp <-
  |
  |#include "TMV_Sym.h"
  |#include <iostream>
  |int main()
  |{
  |  tmv::SymMatrix<double> S(10,4.);
  |  //tmv::Matrix<double> S(10,10,4.);
  |  tmv::Matrix<double> m(10,3,2.);
  |  S += 50.;
  |  tmv::Matrix<double> m2 = m / S;
  |  std::cout<<"23"<<std::endl;
  |  return 0;
  |}
  |
/usr/bin/g++ -o .sconf_temp/conftest_15.o -c -O2 -fno-strict-aliasing -Wall -Werror -Iinclude -I/hom
e/browe/local/64/include .sconf_temp/conftest_15.cpp
/usr/bin/g++ -o .sconf_temp/conftest_15 -Wl,-rpath=#lib -Wl,-rpath=/home/browe/local/64/lib -fopenmp
 .sconf_temp/conftest_15.o -Llib -L/home/browe/local/64/lib -ltmv_symband -lirc -limf -lfftw3 -lpthr
ead -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread
.sconf_temp/conftest_15 > .sconf_temp/conftest_15.out
.sconf_temp/conftest_15: error while loading shared libraries: libmkl_intel_lp64.so: cannot open sha
red object file: No such file or directory

[rmjarvis]

Barney, could you please send me the full gs.error file for master along with the config.log you get for a successful v0.2 compilation?

[rmjarvis]

Oh, unless my recent fix for #312 also fixes this for you, that is. (Although it's less clear to me why this would be the case.)

[barnabytprowe]

Hi Mike, I checked this on the new master and still no dice I'm afraid! Here's a link to the gs.error: https://dl.dropbox.com/u/495780/GalSim/gs.error

And here's the config.log for a successful v0.2 build: https://dl.dropbox.com/u/495780/GalSim/config.log

I'll have a look at the diffs myself to see if I can spot something relevant...

[barnabytprowe]

(Possibly something changed in #307 or #261?)

[rmjarvis]

Could you check if v0.2 passes tests for you? Or even if you can import galsim after building.

Because it looks like the thing that is failing for you is the ability to load the mkl libraries (used by TMV) at run time. We used to not check for this, so it was leading to runtime errors when people would use galsim. Now I put a check in SCons to make sure such a problem is discovered right away, rather than after an apparently successful build. (This was one of the issues in #261.)

This seems to be what is biting you now, so if v0.2 actually worked with this set up, then it means that the test I added isn't right.

But if v0.2 compiled and then failed when doing import galsim, then it means the test is good, but we need to figure out what the right workaround is (and add something to the FAQ). (Advance notice -- one workaround is to do WITH_BLAS=false for TMV. Another is to use CXX=icpc for GalSim. Both should be added to the FAQ.)

[barnabytprowe]

So, I'd thought that v0.2 worked fine. Going back, I actually see that the v0.2 tests almost pass (I didn't notice this originally, to my mild embarrassment):

Starting python tests...
.................................................................................................................................................
----------------------------------------------------------------------
Ran 145 tests in 48.017s

OK
Nosetests finished successfully.

Starting cpp tests...
bin/test_main: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
test_main returned error code  127

GalSim v0.2 also did allow me to import python in interactive mode. So, unfortunately, I'm not sure that the #261 issue you discuss is quite the full story at play here.

For info, tests.log merely contains the info above:

................................................................................................................................
.................
----------------------------------------------------------------------
Ran 145 tests in 48.017s

OK
bin/test_main: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory

However, I'm now suspicious about the tmv-link path itself... I'm not sure it's providing the full path to the actual MKL BLAS and LAPACK libraries. I'll report back soon!

[rmjarvis]

Interesting. The SCons test is more like bin/test_main, since it tries to run an executable. And I thought that if that failed, it would also fail to load via python. So I wonder how python manages to load the mkl library correctly, but the executable doesn't. Hmmm..

One thing you could try is to set EXTRA_LIB_PATH to the directory where libmkl_intel_lp64.so resides.

Also, I see that you have the MKL directories in your LIBRARY_PATH, but not DYLD_LIBRARY_PATH, and there is no LD_LIBRARY_PATH at all. I think LIBRARY_PATH is used at link time to find libraries, and LD_LIBRARY_PATH is used at run time to find shared libraries. (I believe DYLD_LIBRARY_PATH is for Macs only, so I'm not sure that's doing anything for you here.) Anyway, perhaps setting LD_LIBRARY_PATH=LIBRARY_PATH would work.

[rmjarvis]

Correction... you do have LD_LIBRARY_PATH=/home/browe/local/32/lib. So perhaps add either LIBRARY_PATH or just the relevant MKL directory to it.

Also, I have an idea about how SCons might be able to figure this out automatically. I'll push something for you to try...

[rmjarvis]

OK, I just pushed a change where if the previous TMV runtime tests failed, it will try adding (one at a time) each item in your LIBRARY_PATH to the RPATH variable to see if that helps. For your system, I think it will find therein a path with the MKL library that can be set as an RPATH for the executable so it will be found at runtime. Please check if that works for you.

[barnabytprowe]

Hi Mike,

Many thanks for that. Apologies for the delay on my side, unfortunately it didn't yet do the trick. Here's the gs.error: https://dl.dropbox.com/u/495780/GalSim/Builds/gs.error

I tried a few things: adding stuff to the LD_LIBRARY_PATH as you suggested, switching to icpc (and recompiling TMV with icpc for good measure). What I'll do next is advance through the commits on master to see which one causes the installation to break, there aren't too many. I'll let you know when that's done Mike, thanks for all the help so far!

[rmjarvis]

Sorry about that. Your gs.error file pointed out that my attempted fix didn't do what I wanted it to do. Try the one I just pushed.

[barnabytprowe]

Thank you Mike. Interestingly, am now getting some different behaviour with g++ and icpc. In g++, that fixes that test, but then a later test fails: https://dl.dropbox.com/u/495780/GalSim/Builds/gs.error_g%2B%2B_20Nov2012

In icpc, all tests pass, but then there are a load of compilation warnings and some limited errors. The errors are reproduced below:

lucius% scons CXX=icpc WARN=False
scons: Reading SConscript files ...
SCons is version 2.0.1 using python version 2.7.2
Python is from /usr/local/EPD/epd-7.1.2/include/python2.7
Using the following (non-default) scons options:
   CXX = icpc
   WARN = False
   TMV_DIR = /home/browe/local/64
These can be edited directly in the file gs_scons.conf.
Type scons -h for a full list of available options.
Using python =  /usr/bin/env python
Using default PYPREFIX =  /usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages
Using compiler: /usr/local/intel/Compiler/11.1/072/bin/intel64/icpc
compiler version: 11.1
Determined that a good number of jobs = 24
Checking for C++ header file fftw3.h... yes
Checking for correct FFTW linkage... yes
Checking for C++ header file boost/shared_ptr.hpp... yes
Checking for C++ header file TMV.h... yes
Using TMV_LINK file: /home/browe/local/64/share/tmv/tmv-link
     -L/usr/local/intel/Compiler/11.1/072/mkl/lib -L/home/browe/local/64/lib -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread -Wl,-rpath=/home/browe/local/64/lib -openmp
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage... yes
Checking if we can build against Python... yes
Checking if we can build module using TMV... yes
Checking if we can build against NumPy... yes
Checking for PyFITS... yes
Checking if we can build against Boost.Python... yes
scons: done reading SConscript files.
scons: Building targets ...
icpc -o src/.obj/BinomFact.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/BinomFact.cpp
icpc -o src/.obj/FFT.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/FFT.cpp
icpc -o src/.obj/Image.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/Image.cpp
icpc -o src/.obj/Interpolant.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/Interpolant.cpp
icpc -o src/.obj/Laguerre.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/Laguerre.cpp
icpc -o src/.obj/OneDimensionalDeviate.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/OneDimensionalDeviate.cpp
icpc -o src/.obj/PhotonArray.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/PhotonArray.cpp
icpc -o src/.obj/SBProfile.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBProfile.cpp
icpc -o src/.obj/SBBox.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBBox.cpp
icpc -o src/.obj/SBGaussian.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBGaussian.cpp
icpc -o src/.obj/SBExponential.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBExponential.cpp
icpc -o src/.obj/SBSersic.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBSersic.cpp
icpc -o src/.obj/SBMoffat.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBMoffat.cpp
icpc -o src/.obj/SBAiry.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBAiry.cpp
/usr/local/include/c++/4.5.0/iomanip(64): error: expected an expression
    { return { __mask }; }
             ^

/usr/local/include/c++/4.5.0/iomanip(94): error: expected an expression
    { return { __mask }; }
             ^

/usr/local/include/c++/4.5.0/iomanip(125): error: expected an expression
    { return { __base }; }
             ^

/usr/local/include/c++/4.5.0/iomanip(193): error: expected an expression
    { return { __n }; }
             ^

/usr/local/include/c++/4.5.0/iomanip(223): error: expected an expression
    { return { __n }; }
             ^

icpc -o src/.obj/SBLaguerre.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBLaguerre.cpp
icpc -o src/.obj/SBAdd.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBAdd.cpp
icpc -o src/.obj/SBConvolve.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBConvolve.cpp
icpc -o src/.obj/SBTransform.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBTransform.cpp
icpc -o src/.obj/SBInterpolatedImage.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBInterpolatedImage.cpp
icpc -o src/.obj/SBDeconvolve.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBDeconvolve.cpp
icpc -o src/.obj/SBKolmogorov.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/SBKolmogorov.cpp
icpc -o src/.obj/CppShear.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/CppShear.cpp
icpc -o src/.obj/Table.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/Table.cpp
icpc -o src/.obj/RealSpaceConvolve.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/RealSpaceConvolve.cpp
icpc -o src/.obj/Random.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/Random.cpp
icpc -o src/hsm/.obj/PSFCorr.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include src/hsm/PSFCorr.cpp
icpc -o pysrc/.obj/module.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include -I/usr/local/EPD/epd-7.1.2/include/python2.7 -I/usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages/numpy/core/include pysrc/module.cpp
icpc -o pysrc/.obj/Angle.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include -I/usr/local/EPD/epd-7.1.2/include/python2.7 -I/usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages/numpy/core/include pysrc/Angle.cpp
icpc -o pysrc/.obj/Bounds.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include -I/usr/local/EPD/epd-7.1.2/include/python2.7 -I/usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages/numpy/core/include pysrc/Bounds.cpp
icpc -o pysrc/.obj/CppShear.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include -I/usr/local/EPD/epd-7.1.2/include/python2.7 -I/usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages/numpy/core/include pysrc/CppShear.cpp
icpc -o pysrc/.obj/Image.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include -I/usr/local/EPD/epd-7.1.2/include/python2.7 -I/usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages/numpy/core/include pysrc/Image.cpp
icpc -o pysrc/.obj/SBProfile.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include -I/usr/local/EPD/epd-7.1.2/include/python2.7 -I/usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages/numpy/core/include pysrc/SBProfile.cpp
icpc -o pysrc/.obj/SBAdd.os -c -O2 -vec-report0 -fPIC -Iinclude/galsim -Iinclude -I/home/browe/local/64/include -I/usr/local/EPD/epd-7.1.2/include/python2.7 -I/usr/local/EPD/epd-7.1.2/lib/python2.7/site-packages/numpy/core/include pysrc/SBAdd.cpp
compilation aborted for src/Laguerre.cpp (code 2)
scons: *** [src/.obj/Laguerre.os] Error 2
scons: building terminated because of errors.

I have to go to a seminar now for an hour, but I'll try and slyly rebuild TMV without the MKL BLAS... it seems to be causing a lot of bother...

[barnabytprowe]

Hmmmm, I see that part of the problem with the icpc install is that it's using files from /local/include/c++/4.5.0/, which is NOT right. Ugh, installing things on systems where you don't have root is a massive pain!

[barnabytprowe]

OK, so compiling TMV with WITH_BLAS=False and using g++ for both, everything works fine, tests pass. I'm not sure how we want to prioritize getting MKL+TMV+GalSim working on this system, as the issue is now one of extra speed rather than whether or not the code compiles and runs.

[rmjarvis]

Apparently the icpc error is a known bug:

http://software.intel.com/en-us/forums/topic/289446

The supposed work-around is to update to icpc 12 and use an extra flag -std=c++0x. Sounds like going without MKL is the better option.

[rmjarvis]

I'm not sure how we want to prioritize getting MKL+TMV+GalSim working on this system, as the issue is now one of extra speed rather than whether or not the code compiles and runs.

The BLAS stuff will never be a limiting factor for GalSim, so recommending the WITH_BLAS=false option is probably the best way to go. Would you like to put something on the FAQ or should I?

[barnabytprowe]

I'll do it, you've done enough for this issue already Mike, thank you for your help. Once that's done, I think we should close this.

Barnaby Rowe
Postdoctoral Research Associate

Department of Physics & Astronomy
University College London
Gower Street
London WC1E 6BT
United Kingdom

On 20 Nov 2012, at 14:25, Mike Jarvis wrote:

I'm not sure how we want to prioritize getting MKL+TMV+GalSim working on this system, as the issue is now one of extra speed rather than whether or not the code compiles and runs.

The BLAS stuff will never be a limiting factor for GalSim, so recommending the WITH_BLAS=false option is probably the best way to go. Would you like to put something on the FAQ or should I?

—
Reply to this email directly or view it on GitHub.

[barnabytprowe]

OK, have put some info about this in a new item on the FAQ. Closing the issue, thanks for the help Mike.