fftw fails to compile #551

mjakl · 2012-03-09T09:05:32Z

When compiling a recent Julia snapshot, I get this error:

libtool: link: gcc -shared -Wl,--whole-archive kernel/.libs/libkernel.a dft/.libs/libdft.a dft/scalar/.libs/libdft_scalar.a dft/scalar/codelets/.libs/libdft_scalar_codelets.a rdft/.libs/librdft.a rdft/scalar/.libs/librdft_scalar.a rdft/scalar/r2cf/.libs/librdft_scalar_r2cf.a rdft/scalar/r2cb/.libs/librdft_scalar_r2cb.a rdft/scalar/r2r/.libs/librdft_scalar_r2r.a reodft/.libs/libreodft.a api/.libs/libapi.a simd-support/.libs/libsimd_support.a simd-support/.libs/libsimd_sse2_nonportable.a dft/simd/sse2/.libs/libdft_sse2_codelets.a rdft/simd/sse2/.libs/librdft_sse2_codelets.a -Wl,--no-whole-archive -lm -Wl,-rpath -Wl,-soname -Wl,libfftw3f.so.3 -o .libs/libfftw3f.so.3.3.0
/usr/bin/ld: cannot find libfftw3f.so.3: No such file or directory
collect2: ld returned 1 exit status
make[3]: *** [libfftw3f.la] Error 1
make[2]: *** [install-recursive] Error 1
make[1]: *** [/home/JuliaLang/src/julia-build/external/root/lib/libfftw3f.3.so] Error 2
make: *** [julia-release] Error 2

Would it be possible to include a USE_SYSTEM_FFTW for distributions having this problem?

Please also see https://aur.archlinux.org/packages.php?ID=56877

ViralBShah · 2012-03-10T04:17:19Z

Yes, we should have a USE_SYSTEM_FFTW. I'll take a crack at it unless someone beats me to it. :-)

ViralBShah · 2012-03-10T06:00:23Z

Fixed in commit 0d2b254. This was easy. Can you try it out?

mjakl · 2012-03-10T12:48:26Z

FFTW is working fine now. Thank you. Now gmp is having issues, but it seems to be an issues related to Archlinux' build system since issuing the (make) commands directly on the command line works fine. I'll have to dig into that.

ViralBShah · 2012-03-10T13:05:31Z

We should have a USE_SYSTEM_GMP as well.

-viral

On 10-Mar-2012, at 6:18 PM, Michael Jakl wrote:

FFTW is working fine now. Thank you. Now gmp is having issues, but it seems to be an issues related to Archlinux' build system since issuing the (make) commands directly on the command line works fine. I'll have to dig into that.

Reply to this email directly or view it on GitHub:
#551 (comment)

mjakl · 2012-03-10T23:09:51Z

Unfortunately, the USE_SYSTEM_GMP switch does not work. The gmp-wrapper depends on the custom compiled gmp and starts the download/compilation (which in turn fails).

ViralBShah · 2012-03-11T05:07:29Z

Can you try now? Added a few more USE_SYSTEM_* options also.

-viral

On 11-Mar-2012, at 4:39 AM, Michael Jakl wrote:

Unfortunately, the USE_SYSTEM_GMP switch does not work. The gmp-wrapper depends on the custom compiled gmp and starts the download/compilation (which in turn fails).

Reply to this email directly or view it on GitHub:
#551 (comment)

mjakl · 2012-03-11T10:36:49Z

Thank you, the compilation phase works fine now. Yet, the install phase still searches for the libs and fails:

eposix.so lib/librandom.so lib/liblapack.so lib/libsuitesparse*so lib/libgrisu.so lib/libamos.so /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib
install: cannot stat `lib/libarpack.so': No such file or directory
`lib/libfdm.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libfdm.so'
install: cannot stat `lib/libfftw3.so': No such file or directory
install: cannot stat `lib/libfftw3f.so': No such file or directory
install: cannot stat `lib/libpcre.so': No such file or directory
install: cannot stat `lib/libpcrecpp.so': No such file or directory
install: cannot stat `lib/libpcreposix.so': No such file or directory
`lib/librandom.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/librandom.so'
install: cannot stat `lib/liblapack.so': No such file or directory
`lib/libsuitesparse.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse.so'
`lib/libsuitesparse_wrapper.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse_wrapper.so'
`lib/libgrisu.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libgrisu.so'
`lib/libamos.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libamos.so'
make: *** [install] Error 1
==> ERROR: A failure occurred in package().
    Aborting.

I'm using:
USE_SYSTEM_LLVM=1
USE_SYSTEM_READLINE=1
USE_SYSTEM_PCRE=1
USE_SYSTEM_GMP=1
USE_SYSTEM_FFTW=1
USE_SYSTEM_LIBUNWIND=1
USE_SYSTEM_LAPACK=1
USE_SYSTEM_ARPACK=1
USE_SYSTEM_BLAS=1

ViralBShah · 2012-03-11T10:57:09Z

Hmm, I don't think I have tested the installation ever with all these Makefile flags. This will take some work. However, you don't really need to use install if you just copy the relevant files, you should be good to go.

-viral

On 11-Mar-2012, at 4:06 PM, Michael Jakl wrote:

Thank you, the compilation phase works fine now. Yet, the install phase still searches for the libs and fails:

eposix.so lib/librandom.so lib/liblapack.so lib/libsuitesparse_so lib/libgrisu.so lib/libamos.so /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib
install: cannot stat lib/libarpack.so': No such file or directory lib/libfdm.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libfdm.so' install: cannot statlib/libfftw3.so': No such file or directory
install: cannot stat lib/libfftw3f.so': No such file or directory install: cannot statlib/libpcre.so': No such file or directory
install: cannot stat lib/libpcrecpp.so': No such file or directory install: cannot statlib/libpcreposix.so': No such file or directory
lib/librandom.so' ->/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/librandom.so'
install: cannot stat lib/liblapack.so': No such file or directory lib/libsuitesparse.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse.so' lib/libsuitesparse_wrapper.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse_wrapper.so' lib/libgrisu.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libgrisu.so' lib/libamos.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libamos.so'
make: *_* [install] Error 1
==> ERROR: A failure occurred in package().
Aborting.

I'm using:
USE_SYSTEM_LLVM=1
USE_SYSTEM_READLINE=1
USE_SYSTEM_PCRE=1
USE_SYSTEM_GMP=1
USE_SYSTEM_FFTW=1
USE_SYSTEM_LIBUNWIND=1
USE_SYSTEM_LAPACK=1
USE_SYSTEM_ARPACK=1
USE_SYSTEM_BLAS=1

Reply to this email directly or view it on GitHub:
#551 (comment)

mjakl · 2012-03-11T11:31:38Z

I'm trying to get my Archlinux package working again, hence all those USE_SYSTEM flags. As a last resort, I could just copy the files. I'll keep you posted. Thanks for your help!

mjakl · 2012-03-11T11:54:38Z

Replacing line 36 of julia's Makefiles with
>-install -v lib/*.$(SHLIB_EXT) $(DESTDIR)$(PREFIX)/share/julia/lib
works with all those switches.

We need all those libraries in the lib folder, don't we?

ViralBShah · 2012-03-11T12:09:33Z

The simplest thing I can think of is something like:

test -f lib/libfoo.$(SHLIB_EXT) && install -v ...

This doesn't work in the Makefile because for files that are not compiled, test returns an error and the make fails. We need a quick fix for this, and a long term fix for the whole external libraries thing.

-viral

On 11-Mar-2012, at 4:06 PM, Michael Jakl wrote:

Thank you, the compilation phase works fine now. Yet, the install phase still searches for the libs and fails:

eposix.so lib/librandom.so lib/liblapack.so lib/libsuitesparse_so lib/libgrisu.so lib/libamos.so /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib
install: cannot stat lib/libarpack.so': No such file or directory lib/libfdm.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libfdm.so' install: cannot statlib/libfftw3.so': No such file or directory
install: cannot stat lib/libfftw3f.so': No such file or directory install: cannot statlib/libpcre.so': No such file or directory
install: cannot stat lib/libpcrecpp.so': No such file or directory install: cannot statlib/libpcreposix.so': No such file or directory
lib/librandom.so' ->/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/librandom.so'
install: cannot stat lib/liblapack.so': No such file or directory lib/libsuitesparse.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse.so' lib/libsuitesparse_wrapper.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libsuitesparse_wrapper.so' lib/libgrisu.so' -> /home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libgrisu.so' lib/libamos.so' -> `/home/mjakl/PersonalProjects/archlinux-JuliaLang/pkg//usr/share/julia/lib/libamos.so'
make: *_* [install] Error 1
==> ERROR: A failure occurred in package().
Aborting.

I'm using:
USE_SYSTEM_LLVM=1
USE_SYSTEM_READLINE=1
USE_SYSTEM_PCRE=1
USE_SYSTEM_GMP=1
USE_SYSTEM_FFTW=1
USE_SYSTEM_LIBUNWIND=1
USE_SYSTEM_LAPACK=1
USE_SYSTEM_ARPACK=1
USE_SYSTEM_BLAS=1

Reply to this email directly or view it on GitHub:
#551 (comment)

ViralBShah · 2012-03-11T12:10:44Z

Well - we can do without the LLVM libraries, but yes, no harm for the time being.

-viral

On 11-Mar-2012, at 5:24 PM, Michael Jakl wrote:

Replacing line 36 of julia's Makefiles with
> -install -v lib/*.$(SHLIB_EXT) $(DESTDIR)$(PREFIX)/share/julia/lib
Works with all those switches.

We need all those libraries in the lib folder, don't we?

Reply to this email directly or view it on GitHub:
#551 (comment)

ViralBShah · 2012-03-11T12:17:13Z

BTW, just out of curiosity, does Archlinux have lapack, arpack-ng, suitesparse etc? What USE_SYSTEM flags did you end up using? Currently, these are all motivated by what is available on Debian. The only ones not in debian are double-conversion, amos, fdlibm, and dsfmt.

-viral

On 11-Mar-2012, at 5:24 PM, Michael Jakl wrote:

Replacing line 36 of julia's Makefiles with
> -install -v lib/*.$(SHLIB_EXT) $(DESTDIR)$(PREFIX)/share/julia/lib
Works with all those switches.

We need all those libraries in the lib folder, don't we?

Reply to this email directly or view it on GitHub:
#551 (comment)

mjakl · 2012-03-11T12:44:32Z

I'm currently compiling using these flags:

    USE_SYSTEM_LLVM=1 \
    USE_SYSTEM_READLINE=1 \
    USE_SYSTEM_PCRE=1 \
    USE_SYSTEM_GMP=1 \
    USE_SYSTEM_LIBUNWIND=1 \
    USE_SYSTEM_FFTW=1 \
    USE_SYSTEM_LAPACK=1 \
    USE_SYSTEM_ARPACK=1 \
    USE_SYSTEM_BLAS=1

With these dependencies from Archlinux:
depends=('llvm' 'readline' 'pcre' 'gcc-fortran' 'fftw' 'gmp' 'lapack' 'blas' 'arpack' 'libunwind')

The versions are currently:

LLVM 3.0
READLINE 6.2.002
PCRE 8.30
GMP 5.0.4
FORTRAN 4.6.3
LIBUNWIND 1.0.1 (only available via the use repository - AUR)
FFTW 3.3
LAPACK 3.4.0
ARPACK 3.1.0
BLAS 3.4.0

SuiteSparse is also available via AUR, but did not compile on my machine, hence I'm using the one julia compiles itself.

Thank you very much for all your help. I've been able to update the julia-git package on AUR now (https://aur.archlinux.org/packages.php?ID=56877).

mjakl · 2012-03-11T12:48:46Z

I forgot, double-conversion, amos, fdlibm, and dsfmt aren't available on Archlinux too.

ViralBShah · 2012-03-11T13:02:37Z

BTW, looking at the archlinux blas package, it looks like it is using the reference BLAS. Is this the BLAS that julia will also ending up using?

http://www.archlinux.org/packages/extra/i686/blas/

If so, this is likely to be orders of magnitude slower than something like openblas. Unfortunately, currently the blas and lapack in julia are inseparable, and we need some cleanup in the way the libraries are built.

It's a good idea for archlinux to provide openblas, and for julia to depend on that instead at least for platforms that openblas supports. On others, ATLAS may still be faster than reference blas.

-viral

On 11-Mar-2012, at 6:14 PM, Michael Jakl wrote:

I'm currently compiling using these flags:
   USE_SYSTEM_LLVM=1 \
   USE_SYSTEM_READLINE=1 \
   USE_SYSTEM_PCRE=1 \
   USE_SYSTEM_GMP=1 \
   USE_SYSTEM_LIBUNWIND=1 \
   USE_SYSTEM_FFTW=1 \
   USE_SYSTEM_LAPACK=1 \
   USE_SYSTEM_ARPACK=1 \
   USE_SYSTEM_BLAS=1
With these dependencies from Archlinux:
depends=('llvm' 'readline' 'pcre' 'gcc-fortran' 'fftw' 'gmp' 'lapack' 'blas' 'arpack' 'libunwind')

The versions are currently:

LLVM 3.0

READLINE 6.2.002

PCRE 8.30

GMP 5.0.4

FORTRAN 4.6.3

LIBUNWIND 1.0.1 (only available via the use repository - AUR)

FFTW 3.3

LAPACK 3.4.0

ARPACK 3.1.0

BLAS 3.4.0

SuiteSparse is also available via AUR, but did not compile on my machine, hence I'm using the one julia compiles itself.

Thank you very much for all your help. I've been able to update the julia-git package on AUR now (https://aur.archlinux.org/packages.php?ID=56877).

Reply to this email directly or view it on GitHub:
#551 (comment)

mjakl · 2012-03-11T13:17:14Z

I see. The BLAS package would indeed be the reference implementation. I've reverted back to using lapack and openblas provided by julia itself.
Thanks again :)

ViralBShah · 2012-03-11T13:26:46Z

What happens with ARPACK? Would it link to the Archlinux ARPACK, LAPACK, and BLAS? Sorry, all the stuff that links to BLAS and LAPACK is a bit of a mess right now.

-viral

On 11-Mar-2012, at 6:47 PM, Michael Jakl wrote:

I see. The BLAS package would indeed be the reference implementation. I've reverted back to using lapack and openblas provided by julia itself.
Thanks again :)

Reply to this email directly or view it on GitHub:
#551 (comment)

ViralBShah · 2012-03-11T13:30:41Z

It should be ok to use the system arpack, which uses the system lapack and blas for now, because arpack doesn't seem to use level 3 BLAS.

Eventually, though, it's best to have Archlinux include openblas.

-viral

On 11-Mar-2012, at 6:47 PM, Michael Jakl wrote:

I see. The BLAS package would indeed be the reference implementation. I've reverted back to using lapack and openblas provided by julia itself.
Thanks again :)

Reply to this email directly or view it on GitHub:
#551 (comment)

mjakl · 2012-03-11T14:12:22Z

I'd like to integrate julia nicely into the system, but it seems that I'd need to create my own set of arpack, lapack, openblas libs, I think it's OK to use the custom ones directly in julia where necessary.

ViralBShah · 2012-03-11T15:37:02Z

I think that is a good idea. We will have more packages that will depend on BLAS. CLP is in the works somewhere, and then there is SuiteSparse already that needs a good BLAS.

-viral

On 11-Mar-2012, at 7:42 PM, Michael Jakl wrote:

I see, I'll look into creating a fitting OpenBLAS package. In the meantime I'll leave arpack as system library.

Reply to this email directly or view it on GitHub:
#551 (comment)

also optimize `getproperty` where first argument is a constant module

ghost assigned ViralBShah Mar 10, 2012

ViralBShah closed this as completed Mar 10, 2012

ViralBShah added a commit that referenced this issue Mar 10, 2012

Allow USE_SYSTEM_GMP (#551)

fe65d7c

ViralBShah added a commit that referenced this issue Mar 11, 2012

Quick fix for installing external libraries as in #551

e49ca90

JeffBezanson mentioned this issue Mar 19, 2012

shared library names and search path #353

Closed

Keno pushed a commit that referenced this issue Oct 9, 2023

Merge pull request #551 from JuliaDebug/kc/getproperty_optimize

f130511

also optimize `getproperty` where first argument is a constant module

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fftw fails to compile #551

fftw fails to compile #551

mjakl commented Mar 9, 2012

ViralBShah commented Mar 10, 2012

ViralBShah commented Mar 10, 2012

mjakl commented Mar 10, 2012

ViralBShah commented Mar 10, 2012

mjakl commented Mar 10, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

fftw fails to compile #551

fftw fails to compile #551

Comments

mjakl commented Mar 9, 2012

ViralBShah commented Mar 10, 2012

ViralBShah commented Mar 10, 2012

mjakl commented Mar 10, 2012

ViralBShah commented Mar 10, 2012

mjakl commented Mar 10, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

ViralBShah commented Mar 11, 2012

mjakl commented Mar 11, 2012

ViralBShah commented Mar 11, 2012