Broader platform support #150

Alexander-Barth · 2021-11-30T16:50:43Z

It would be great if NCDatasets.jl could be run on all platforms on which julia is supported. However, currently only on these platforms we have a NetCDF binary:

 NetCDF.v400.702.400.aarch64-linux-gnu.tar.gz
 NetCDF.v400.702.400.i686-w64-mingw32.tar.gz
 NetCDF.v400.702.400.x86_64-apple-darwin.tar.gz
 NetCDF.v400.702.400.x86_64-linux-gnu.tar.gz
 NetCDF.v400.702.400.x86_64-w64-mingw32.tar.gz

Installation of NCDatasets.jl will fail on other platforms.
Notably, this leaves out Raspberry PIs (32-bit armv7l), Linux on PowerPC, FreeBSD, Linux with musl, Apple M1,...

To support more architectures the NetCDF_jll package and HDF5_jll package would need to be updated. As a first test, it would be useful to know if native compilation (of the NetCDF sources) on these platforms pass all the tests (make check) and then attempt cross-compilation (using BinaryBuilder).

ref #116

The text was updated successfully, but these errors were encountered:

visr · 2021-11-30T17:44:02Z

Regarding cross-compilation, after tracing the breadcrumbs again it seems we now have an upstream issue we can refer to: HDFGroup/hdf5#1203.

truedichotomy · 2021-12-02T07:03:32Z

I'm very interested having NCDatasets.jl support Apple Silicon. I just installed netcdf 4.8.1 and HDF5 1.12.1 via homebrew on my M1 mac and they seem to function fine without having done too deep testing.

https://formulae.brew.sh/formula/hdf5
https://formulae.brew.sh/formula/netcdf

Alexander-Barth · 2021-12-02T13:16:04Z

The first step would be to make a PR to update these build instructions to support Apple M1 (or any new architecture):

https://github.com/JuliaPackaging/Yggdrasil/blob/master/H/HDF5/build_tarballs.jl

So far, it uses HDF5 from python on MacOS x86_64.

truedichotomy · 2021-12-04T02:58:12Z

I proposed some changes, but I don't know how to test whether they work.

Alexander-Barth · 2021-12-04T21:16:27Z

One should be able to test it with this command (https://docs.binarybuilder.org/stable/):

julia --color=yes build_tarballs.jl --verbose

You would need to install docker. Maybe you need to run this using the x86_64 emulator. It should create a tar.gz file for your architecture. When you decompress it, it should contain the library for your system.

Maybe this is helpful to force Julia to use a custom library (for HDF5 and NetCDF):

#126 (comment)

truedichotomy · 2021-12-09T04:56:20Z

In my miniforge lib directory I saw the following two netcdf library files: libnetcdf.19.dylib and libnetcdf.a. Would either of them work? I also tried to follow the steps you mentioned for forcing Julia to use custom library. When I tried it with libnetcdf.19.dylib, I'm still getting libnetcdf not defined error when using NCDatasets. Is there anything extra I need to do to make NCDatasets see the change in libnetcdf?

Alexander-Barth · 2021-12-10T10:42:33Z

In Linux libnetcdf.a, is a static library (which will not work, as we need a dynamic library). I guess this is true of also for Mac OS. So I would go with libnetcdf.19.dylib. Did you restart the julia session after calling set_preferences! ? Maybe you can include all the steps that lead to the error with the full error message as a comment here?

The libraries on which netcdf depends (HDF5, zlib, ...) can also be a source of problem. The promise of Yggdrasil is to compile/prepare all the libraries so ensure their comparability otherwise there are problems to similar to the issue described here:
JuliaIO/ImageMagick.jl#140

Alexander-Barth · 2021-12-10T10:50:52Z

Related issue for reference:
h5py/h5py#2008

truedichotomy · 2022-02-23T06:00:22Z

With the latest NetCDF_jll version (v400.702.402+0) and HDF5_jll (v1.12.1+0), I'm able to use NCDatasets.jl on my Apple Silicon M1 mac without the libnetcdf not defined error.

Alexander-Barth · 2022-02-23T15:11:29Z

With the latest NetCDF_jll version (v400.702.402+0) and HDF5_jll (v1.12.1+0), I'm able to use NCDatasets.jl on my Apple Silicon M1 mac without the libnetcdf not defined error.

This is great to know! Thanks a lot @felixcremer for implementing this!

https://github.com/JuliaPackaging/Yggdrasil/pull/4481/files

claresinger · 2022-05-02T18:34:53Z

Hi. I'm having some trouble getting NCDatasets to work on my M1 MacBook air. When I add NCDatasets it does not use the latest versions of NETCDF_jll and HDF5_jll, which I guess is the problem. But when I try to force it to use these versions (NetCDF_jll v400.702.402+0 and HDF5_jll v1.12.1+0) I get errors that the versions are not compatible with what NCDatasets is looking for. Any help? Below is output from my REPL. I am using julia v1.7.2 (2022-02-06).

(experiments) pkg> add NCDatasets
   Resolving package versions...
    Updating `~/Research/MixedLayerModel.jl/experiments/Project.toml`
  [85f8d34a] + NCDatasets v0.12.4
    Updating `~/Research/MixedLayerModel.jl/experiments/Manifest.toml`
  [179af706] + CFTime v0.1.2
  [85f8d34a] + NCDatasets v0.12.4
  [0234f1f7] + HDF5_jll v1.12.0+1
  [7243133f] + NetCDF_jll v400.702.400+0

Alexander-Barth · 2022-05-03T09:01:33Z

As a test (or work-around) can you install version 0.11.9 of NCDatasets?

]add NCDatasets@0.11.9

Does this work?

It is kind of a struggle right now to support Windows and Mac OS M1 as there is no NetCDF_jll version which runs on both and dependencies must be platform independent for Julia's package manager. The real fix would be to make the latest version of NetCDF_jll available on Windows.

claresinger · 2022-05-03T16:25:23Z

As a test (or work-around) can you install version 0.11.9 of NCDatasets?
]add NCDatasets@0.11.9
Does this work?

Yes thank you, this does work! I hope the platform support of NetCDF_jll is improved in the future so this isn't so annoying for you to maintain!

navidcy · 2022-11-02T04:36:20Z

Not sure if this is related but seems like NCDatasets can't be used on Linux with x86 architecture + Julia 1.8. Am I right?

navidcon@satori-login-001:~/test_project$ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.2 (2022-09-29)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |

(@v1.8) pkg> activate .
  Activating new project at `~/test_project`

(test_project) pkg> add NCDatasets
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/test_project/Project.toml`
  [85f8d34a] + NCDatasets v0.12.8
    Updating `~/test_project/Manifest.toml`
  [179af706] + CFTime v0.1.2
  [34da2185] + Compat v4.3.0
  [864edb3b] + DataStructures v0.18.13
  [692b3bcd] + JLLWrappers v1.4.1
  [85f8d34a] + NCDatasets v0.12.8
  [bac558e1] + OrderedCollections v1.4.1
  [21216c6a] + Preferences v1.3.0
  [0234f1f7] + HDF5_jll v1.12.2+2
  [94ce4f54] + Libiconv_jll v1.16.1+1
  [7243133f] + NetCDF_jll v400.902.5+1
  [458c3c95] + OpenSSL_jll v1.1.17+0
  [02c8fc9c] + XML2_jll v2.9.14+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.8.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.1
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v0.5.2+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [4536629a] + OpenBLAS_jll v0.3.20+0
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850b90] + libblastrampoline_jll v5.1.1+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0

julia> using NCDatasets
ERROR: InitError: UndefVarError: libnetcdf not defined
Stacktrace:
  [1] nc_inq_libvers
    @ ~/.julia/packages/NCDatasets/ipGBH/src/netcdf_c.jl:242 [inlined]
  [2] netcdf_version()
    @ NCDatasets ~/.julia/packages/NCDatasets/ipGBH/src/netcdf_c.jl:2157
  [3] init_certificate_authority()
    @ NCDatasets ~/.julia/packages/NCDatasets/ipGBH/src/netcdf_c.jl:2170
  [4] __init__()
    @ NCDatasets ~/.julia/packages/NCDatasets/ipGBH/src/NCDatasets.jl:33
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base ./loading.jl:831
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
    @ Base ./loading.jl:1039
  [7] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1315
  [8] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [9] macro expansion
    @ ./loading.jl:1180 [inlined]
 [10] macro expansion
    @ ./lock.jl:223 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
during initialization of module NCDatasets

(test_project) pkg> st
Status `~/test_project/Project.toml`
  [85f8d34a] NCDatasets v0.12.8

julia> versioninfo()
Julia Version 1.8.2
Commit 36034abf26 (2022-09-29 15:21 UTC)
Platform Info:
  OS: Linux (ppc64le-redhat-linux)
  CPU: 128 × POWER9, altivec supported
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, pwr9)
  Threads: 24 on 128 virtual cores
Environment:
  LD_LIBRARY_PATH = /software/cuda/11.4/lib:/software/cuda/11.4/lib64
  JULIA_DEPOT_PATH = /home/navidcon/.julia:/share/julia/site/
  JULIA_CUDA_USE_BINARYBUILDER = false
  JULIA_LOAD_PATH = @:@v#.#:@stdlib:@site
  JULIA_NUM_THREADS = 24

julia> Sys.ARCH
:powerpc64le

Julia>

Alexander-Barth · 2022-11-02T07:23:06Z

Yes, this is the PowerPC architecture (x86 architecture referrers generally to the Intel/AMD CPU architecture, which is supported in its 64-bit variant). As a first step, we would need to repack HDF5 for PowerPC which seems to be available (https://anaconda.org/anaconda/hdf5/files). Can somebody interested in PowerPC make a PR to HDF5_jll ?

Alexander-Barth · 2023-06-02T14:44:51Z

The supported platforms of NetCDF_jll has been recently significantly extended:

https://github.com/JuliaBinaryWrappers/NetCDF_jll.jl/releases/tag/NetCDF-v400.902.208%2B0

Alexander-Barth · 2023-12-18T21:25:12Z

I am closing this issue as the platform support has significantly improved thanks to these changes in HDF5:
JuliaPackaging/Yggdrasil@bf78a45

Alexander-Barth mentioned this issue Dec 20, 2021

ERROR: LoadError: UndefVarError: libnetcdf not defined #152

Closed

samhatfield mentioned this issue May 17, 2022

Out-of-date NetCDF_jjl and HDF5_jjl for M1 Mac JuliaGeo/NetCDF.jl#157

Closed

Alexander-Barth mentioned this issue Jul 8, 2022

NetCDF_jll broken on v1.8.0-rc1 on macOS and linux #182

Closed

This was referenced Jul 25, 2022

Unable to push [compat] for NCDatasets to 0.12 because of NetCDF_jll GeoRegionsEcosystem/NASAPrecipitation.jl#7

Closed

Unable to push [compat] of NCDatasets.jl to 0.12 GeoRegionsEcosystem/ERA5Reanalysis.jl#7

Closed

This was referenced Nov 2, 2022

Distributed ensemble simulations CliMA/ParameterEstimocean.jl#295

Merged

CATKE and GM in a near-global 1 degree simulation + updates Ri-based vertical diffusivity CliMA/Oceananigans.jl#2786

Merged

navidcy mentioned this issue Dec 7, 2022

Fix bug loading netcdf CliMA/Oceananigans.jl#2847

Merged

glwagner mentioned this issue Dec 8, 2022

Check if NetCDF_jll is_available before loading code #193

Merged

Alexander-Barth closed this as completed Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Broader platform support #150

Broader platform support #150

Alexander-Barth commented Nov 30, 2021 •

edited

Loading

visr commented Nov 30, 2021

truedichotomy commented Dec 2, 2021 •

edited

Loading

Alexander-Barth commented Dec 2, 2021 •

edited

Loading

truedichotomy commented Dec 4, 2021

Alexander-Barth commented Dec 4, 2021

truedichotomy commented Dec 9, 2021 •

edited

Loading

Alexander-Barth commented Dec 10, 2021

Alexander-Barth commented Dec 10, 2021

truedichotomy commented Feb 23, 2022

Alexander-Barth commented Feb 23, 2022

claresinger commented May 2, 2022 •

edited

Loading

Alexander-Barth commented May 3, 2022

claresinger commented May 3, 2022

navidcy commented Nov 2, 2022

Alexander-Barth commented Nov 2, 2022 •

edited

Loading

Alexander-Barth commented Jun 2, 2023 •

edited

Loading

Alexander-Barth commented Dec 18, 2023

Broader platform support #150

Broader platform support #150

Comments

Alexander-Barth commented Nov 30, 2021 • edited Loading

visr commented Nov 30, 2021

truedichotomy commented Dec 2, 2021 • edited Loading

Alexander-Barth commented Dec 2, 2021 • edited Loading

truedichotomy commented Dec 4, 2021

Alexander-Barth commented Dec 4, 2021

truedichotomy commented Dec 9, 2021 • edited Loading

Alexander-Barth commented Dec 10, 2021

Alexander-Barth commented Dec 10, 2021

truedichotomy commented Feb 23, 2022

Alexander-Barth commented Feb 23, 2022

claresinger commented May 2, 2022 • edited Loading

Alexander-Barth commented May 3, 2022

claresinger commented May 3, 2022

navidcy commented Nov 2, 2022

Alexander-Barth commented Nov 2, 2022 • edited Loading

Alexander-Barth commented Jun 2, 2023 • edited Loading

Alexander-Barth commented Dec 18, 2023

Alexander-Barth commented Nov 30, 2021 •

edited

Loading

truedichotomy commented Dec 2, 2021 •

edited

Loading

Alexander-Barth commented Dec 2, 2021 •

edited

Loading

truedichotomy commented Dec 9, 2021 •

edited

Loading

claresinger commented May 2, 2022 •

edited

Loading

Alexander-Barth commented Nov 2, 2022 •

edited

Loading

Alexander-Barth commented Jun 2, 2023 •

edited

Loading