Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDF5 support for Raspberry Pi #794

Closed
timothyjward opened this issue Oct 1, 2019 · 17 comments
Closed

HDF5 support for Raspberry Pi #794

timothyjward opened this issue Oct 1, 2019 · 17 comments
Assignees

Comments

@timothyjward
Copy link

I've noticed that there is no version of the HDF5 preset for armhf. The preset mentions missing support for cross-compilation, but this was changed several years ago, and there are now Debian packages for HDF5 on armhf which means that it should be fine to run on Raspbian. I'm therefore wondering if it would be possible to start producing the preset for this platform?

@saudet
Copy link
Member

saudet commented Oct 1, 2019

Yes, we just need to get these files from their code generator running on ARM devices:
https://github.com/bytedeco/javacpp-presets/blob/master/hdf5/hdf5-linux-ppc64le.patch
I think we can use QEMU for that instead of having the hardware. I just haven't gotten around to it yet.

@peardox
Copy link

peardox commented Oct 4, 2019

Just get Raspberry to send you kit - easy enough

@peardox
Copy link

peardox commented Oct 4, 2019

Balenea https://www.balena.io/blog/balena-releases-first-fully-functional-64-bit-os-for-the-raspberry-pi-4/ have just released a Yocto based 'runs on all pi, 32 + 64' release. This is based on Yocto - https://www.yoctoproject.org.

The point here is that it's all container based so may make an interesting option with less compatibility issues.

@saudet
Copy link
Member

saudet commented Oct 5, 2019

@peardox We still need to run and maintain the hardware somehow. I don't have the resources to do that all by myself. Although, for this particular issue, would you be able to get us the generated files that we need to cross compile HDF5? That would help to get CI going.

@peardox
Copy link

peardox commented Oct 5, 2019

Which ones you after?

ARM64 + ARMHF I guess. No problem. Tell me exactly what you need and that's what I'll create.

ARMHF will take a few days - the best one's out on loan, got another though that need some small fixes I may be able to get running sooner.

I notice you've got a hardware request section on bytedeco.org. I can look into getting you some hardware and cloud servers if you like. Sourcing (for free) was part of my job 20 years ago. Raspberry should be easy as should NVidia Rock etc (All ARM). Cloud server would be worth trying for PPC64le + Power8/9 (may be able to get physical kit for power 8)

@saudet
Copy link
Member

saudet commented Oct 5, 2019

Thanks! We need the 2 files shown in the patch file: H5lib_settings.c and H5Tinit.c, for both ARMHF and ARM64 yes.

Having hardware and/or cloud servers is good, thanks a lot, but we'd still need someone to setup Jenkins or something. It might be possible to use Travis CI on those machines, but I wouldn't count on it, it's probably easier to just use QEMU. Anyway, someone would need to try it out:
https://enterprise.travis-ci.com/

@peardox
Copy link

peardox commented Oct 5, 2019 via email

@peardox
Copy link

peardox commented Oct 6, 2019

HDF5 ARM64 builds fine on Jetson Nano - see debug

The files H5lib_settings.c and H5Tinit.c are nowhere on the system - looks like nano's pre-patched

Note libhdf5-dev not installed for this run - see apt list

Going to try unofficial Pi4 build on ARM64 with known issues pre-fixed (flycapture, libssl0dev + jbig) as this is likely fresher (beta) than the Nano OS.

After that I have a 60G SD ready to test stock Pi4. Something occurred to me while nodding off to sleep - Pi default to a stupid sway size (100M) so gonna give it stupid swap in other direction (8g, both 32 + 64 bit versions of OS)

=================== apt list ==================
hdf5-helpers/bionic,now 1.10.0-patch1+docs-4 arm64 [installed,auto-removable]
hdf5-tools/bionic,now 1.10.0-patch1+docs-4 arm64 [installed]
libhdf5-100/bionic,now 1.10.0-patch1+docs-4 arm64 [installed,automatic]
libhdf5-cpp-100/bionic,now 1.10.0-patch1+docs-4 arm64 [installed,auto-removable]
libhdf5-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-doc/bionic 1.10.0-patch1+docs-4 all
libhdf5-java/bionic 1.10.0-patch1+docs-4 all
libhdf5-jni/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-mpi-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-mpich-100/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-mpich-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-openmpi-100/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-openmpi-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-serial-dev/bionic 1.10.0-patch1+docs-4 all
libjhdf5-java/bionic 2.11.0+dfsg-3 arm64
libjhdf5-jni/bionic 2.11.0+dfsg-3 arm64
libpdl-io-hdf5-perl/bionic 1:0.73-3build2 arm64
python-hdf5storage/bionic 0.1.14-1 all
python-hdf5storage-doc/bionic 0.1.14-1 all
python3-hdf5storage/bionic 0.1.14-1 all
r-cran-hdf5/bionic 1.6.10-4.1 arm64
yorick-hdf5/bionic 0.8.0-8 arm64
=================== debug ==================
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] JavaCPP Presets for CPython 3.7.4-1.5.2-SNAPSHOT ... SUCCESS [15:51 min]
[INFO] JavaCPP Presets for NumPy 1.17.2-1.5.2-SNAPSHOT .... SUCCESS [10:38 min]
[INFO] JavaCPP Presets for HDF5 1.10.5-1.5.2-SNAPSHOT ..... SUCCESS [ 20.954 s]
[INFO] JavaCPP Presets for LLVM 9.0.0-1.5.2-SNAPSHOT ...... SUCCESS [05:06 min]
[INFO] JavaCPP Presets for Leptonica 1.78.0-1.5.2-SNAPSHOT SUCCESS [08:03 min]
[INFO] JavaCPP Presets for Tesseract 4.1.0-1.5.2-SNAPSHOT . SUCCESS [06:51 min]
[INFO] JavaCPP Presets for cpu_features 0.4.1-1.5.2-SNAPSHOT SUCCESS [ 14.842 s]
[INFO] JavaCPP Presets for Systems 1.5.2-SNAPSHOT ......... FAILURE [ 4.619 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47:14 min
[INFO] Finished at: 2019-10-06T07:49:24+01:00

@saudet
Copy link
Member

saudet commented Oct 9, 2019

You'll need to build HDF5 from source to get those generated files.

@peardox
Copy link

peardox commented Oct 11, 2019

OK - I'll give it a go - hdf5-1.10.5.tar.bz2 as per hdf5/cppbuild.sh I presume

Use any flags or just ...

./configure
make

Here's the output of ./configure

General Information:
-------------------
                   HDF5 Version: 1.10.5
                  Configured on: Fri Oct 11 01:28:46 BST 2019
                  Configured by: simon@nano
                    Host system: aarch64-unknown-linux-gnu
              Uname information: Linux nano 4.9.140-tegra #1 SMP PREEMPT Tue Sep 10 14:23:34 BST 2019 aarch64 aarch64 aarch64 GNU/Linux
                       Byte sex: little-endian
             Installation point: /home/simon/src/hdf5-1.10.5/hdf5

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /usr/bin/gcc
                       CPPFLAGS:
                    H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS:
                        C Flags:
                     H5 C Flags:  -std=c99  -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: no

                           Java: no


Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                    Threadsafety: no
             Default API mapping: v110
  With deprecated public symbols: yes
          I/O filters (external): deflate(zlib)
                             MPE: no
                      Direct VFD: no
                         dmalloc: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
       Strict file format checks: no
    Optimization instrumentation: no

@saudet
Copy link
Member

saudet commented Oct 11, 2019 via email

@peardox
Copy link

peardox commented Oct 11, 2019

OK - I'll do arm64 + 32 in the AM

@saudet
Copy link
Member

saudet commented Oct 11, 2019

Although, strangely enough, H5lib_settings.c contains information about the build configuration. Might be best to use the same flags as used for other architectures in cppbuild.sh.

@peardox
Copy link

peardox commented Oct 11, 2019 via email

@peardox
Copy link

peardox commented Oct 11, 2019

armhf = Raspberry Pi4 + Raspbian (Debian Buster)
arm64 = Jetson Nano + Ubuntu 18.04

See relevant hdf5--settings.log for build settings for configure switches in cppbuild.sh mode
Each arch has asis (straight configure) and cppbuild dir (estimated cppbuild configure for arm64)

hdf5.zip

saudet added a commit that referenced this issue Oct 12, 2019
 * Add `linux-armhf` and `linux-arm64` builds for HDF5 (issue #794)
@saudet
Copy link
Member

saudet commented Oct 13, 2019

Ok, it's done! Let me know if you encounter any issues with the snapshots: http://bytedeco.org/builds/

@saudet
Copy link
Member

saudet commented Nov 6, 2019

And binaries have now been released with version 1.5.2! Enjoy and thanks for the contributions

@saudet saudet closed this as completed Nov 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants