Latest FFTW release causes portability issues

[bradcarman]

It appears the code changes in FFTW.jl for v1.4.3 causes hard coded paths to be compiled when using PackageCompiler.jl causing this package to no longer be portable

Here is where I think the problem is occurring...

    @static if fftw_provider == "fftw"
        fftw_init_threads()
        libfftw3[] = FFTW_jll.libfftw3_path
        libfftw3f[] = FFTW_jll.libfftw3f_path
    end
    @static if fftw_provider == "mkl"
        libfftw3[] = MKL_jll.libmkl_rt_path
        libfftw3f[] = MKL_jll.libmkl_rt_path
    end

[giordano]

Related to #208

[mkborregaard]

Could this be what I'm experiencing? I cannot install FFTW, making it impossible to e.g. run Makie.

julia> using FFTW
[ Info: Precompiling FFTW [7a1cc6ca-52ef-59f5-83cd-3a7055c09341]
ERROR: LoadError: UndefVarError: libfftw3_path not defined
Stacktrace:
  [1] getproperty(x::Module, f::Symbol)
    @ Base ./Base.jl:35
  [2] top-level scope
    @ ~/.julia/packages/FFTW/kKdEk/src/providers.jl:51
  [3] include(mod::Module, _path::String)
    @ Base ./Base.jl:417
  [4] include(x::String)
    @ FFTW ~/.julia/packages/FFTW/kKdEk/src/FFTW.jl:1
  [5] top-level scope
    @ ~/.julia/packages/FFTW/kKdEk/src/FFTW.jl:17
  [6] include
    @ ./Base.jl:417 [inlined]
  [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1318
  [8] top-level scope
    @ none:1
  [9] eval
    @ ./boot.jl:373 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [11] top-level scope
    @ none:1
in expression starting at /Users/cvg147/.julia/packages/FFTW/kKdEk/src/providers.jl:49
in expression starting at /Users/cvg147/.julia/packages/FFTW/kKdEk/src/FFTW.jl:1
ERROR: Failed to precompile FFTW [7a1cc6ca-52ef-59f5-83cd-3a7055c09341] to /Users/cvg147/.julia/compiled/v1.7/FFTW/jl_BMQSwy.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
   @ Base ./loading.jl:1466
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1410
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1120
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997

My system:

julia> versioninfo()
Julia Version 1.7.0-beta3
Commit e76c9dad42 (2021-07-07 08:12 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin20.5.0)
  CPU: Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.0 (ORCJIT, cyclone)

(@v1.7) pkg> st
      Status `~/.julia/environments/v1.7/Project.toml`
  [7a1cc6ca] FFTW v1.4.3
  [e9467ef8] GLMakie v0.4.6

[jeremiahpslewis]

@mkborregaard I have the same issue (and same setup M1 Mac + Julia 1.7). I don't run into any problems on Julia 1.6.

The issue looks like it might be with how FFTW_jll is loaded by Julia 1.7 (potentially only on M1 Macs?).

It doesn't seem to be an issue with the FFTW_jll binary itself, as the same binary version, v3.3.9+8, is used by Julia 1.6, where it works, and Julia 1.7, where it doesn't.

Do you think it might be worth reporting this as a regression in the 1.7 release or is there another likely cause?

[mkborregaard]

Yes it sounds reasonable to report this as a regression

[giordano]

@mkborregaard @jeremiahpslewis

1. What you're seeing does not have anything to do with the present issue nor Problem with FFTW=1.4.3 ignoring DEPOT_PATH variable at runtime #208
2. What you're observing is that FFTW_jll doesn't have a build for your platform, which was already reported in Darwin ARM: libfftw3 not defined #187
3. In particular, a solution to your problem was already offered in Darwin ARM: libfftw3 not defined #187 (comment)
4. This isn't a regression in Julia, the issue you opened should be closed
5. You'll find many other libraries missing for your platform, see Meta issue about building packages for experimental platforms (aarch64-apple-darwin and armv6l-linux-*) JuliaPackaging/Yggdrasil#2763. Please don't open new issues every time you bump into a missing library in a jll package
6. Be aware you'll have some other issues on that platform (it's currently tier 3), including random segfaults, see Darwin/ARM64 tracking issue JuliaLang/julia#36617 and https://github.com/JuliaLang/julia/labels/apple%20silicon

[jeremiahpslewis]

@giordano Thank you for the clarification; this sounds like a frustrating issue to manage from a community perspective and one that comes up a lot. To be fair, it seems like having some level of testing by users, even of experimental platforms, is a necessary and useful prerequisite for moving these platforms toward full support.

One thought: would it be possible / reasonable for Julia to throw a warning when it tries to load a _jll build which is not optimized for a specific platform / when the build is not available for the platform in question? It is counterintuitive to see that the package can be added to the environment and then to have it fail only afterwards because of incompatibility. I wonder whether this 'silent failure' behavior is generating unnecessary 'issue noise' from myself and others.

Thanks again for the clear and extensive response and for your work on binaries which is so key to a smooth Julia experience!

[mkborregaard]

Thanks @giordano
I'm a bit surprised that Mac (at least almost all current hardware) is considered an "experimental platform" from the JuliaLang perspective. I'm working on the standard mac provided by my institution, and mac presently has 15.6% of the global OS market share.

[mkborregaard]

But I guess that just goes to show that Apple consistently makes it hard for open source - I do understand the frustration associated with maintaining for this.

[jpsamaroo]

@mkborregaard M1 Macs are not the same as x86_64 Macs, and they are different in many ways (specifically using an entirely different CPU ISA, among other things) which makes supporting them in JLLs non-trivial. Give the community some time and they will get JLLs working on M1s.

[stevengj]

Duplicate of #187?

[mkborregaard]

@jpsamaroo sure thing, to be clear I'm not blaming the community for not being up to speed with every new technology. I was just genuinely surprised.

[StefanKarpinski]

M1 is an entirely new toolchain for everyone — LLVM, gfortran, gcc, etc. It took the better part of the past year for gfortran to support M1, without which basically nothing numerical works. Now there are still LLVM issues with the platform as well as other compiler problems.

[mkborregaard]

Most of the discussion here, initiated by me (sorry) seems to be a duplicate of #187 , but the original post actually isn't. Maybe @bradcarman wants to close this and create a fresh issue? Sorry for noise.

[KristofferC]

I think this should be fixed in the new release.

[mkborregaard]

I still have the exact same issue (#210 (comment)) on master and v1.4.4.

[KristofferC]

Yes, but that is unrelated to this issue. I am not really sure why that is getting discussed here further after the explanation in #210 (comment).

[mkborregaard]

OK fair enough, I misunderstood what you meant by "this".

[KristofferC]

I'll close this since I believe it is fixed, feel free to comment otherwise.