Issues installing BPCells, unable to identify root error

[pravinvel98]

Hi I have been trying to install BPCells unsuccessfully for some time now, I am unable to figure out the exact root cause of the issue. The error seems to stem from some issues with the loaded hdf5 library. The hdf5 library appears to be functional but the highway.h library is not compiled correctly. Attached below is the verbose log. Kindly let me know what the exact issue could be or any suggestions for troubleshooting. Thanks a lot.

remotes::install_github("bnprks/BPCells/r")
Downloading GitHub repo bnprks/BPCells@HEAD
── R CMD build ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ checking for file ‘/local/tmp/RtmpRcff7x/remotes42af69ba0e19/bnprks-BPCells-290863d/r/DESCRIPTION’ (1.1s)
─ preparing ‘BPCells’:
✔ checking DESCRIPTION meta-information ...
─ cleaning src
─ running ‘cleanup’
─ checking for LF line-endings in source and make files and shell scripts (2.2s)
─ checking for empty or unneeded directories (369ms)
─ building ‘BPCells_0.2.0.tar.gz’
Warning: invalid uid value replaced by that for user 'nobody'

Installing package into ‘/omics/odcf/analysis/OE0415_projects_temp/nb/pravin_temp/R_temp’
(as ‘lib’ is unspecified)

• installing source package ‘BPCells’ ...
  ** using staged installation

• '[' -z '' ']'
• ENABLE_INSTALL_COUNTING=yes
• '[' yes == yes ']'
• curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure
• echo 'Recording install count metrics'
  Recording install count metrics
  ++ /software/r/4.4.0/lib64/R/bin/R CMD config CC
• CC=/software/gcc/12.2.0/bin/gcc
  ++ /software/r/4.4.0/lib64/R/bin/R CMD config CXX
• CXX='/software/gcc/12.2.0/bin/g++ -std=c++17'
• ENV_CFLAGS='-g -O2'
• ENV_LDFLAGS=-L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib
  ++ /software/r/4.4.0/lib64/R/bin/R CMD config CFLAGS
• CFLAGS='-g -O2 -g -O2'
  ++ /software/r/4.4.0/lib64/R/bin/R CMD config CXXFLAGS
• CXXFLAGS='-g -O2 -g -O2'
  ++ /software/r/4.4.0/lib64/R/bin/R CMD config LDFLAGS
• LDFLAGS='-L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib -L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib'
• echo 'Testing hdf5 by compiling example program...'
  Testing hdf5 by compiling example program...
• HDF5_CFLAGS=
• HDF5_LIBS=-lhdf5
• HDF5_OK=
• /software/gcc/12.2.0/bin/gcc tools/h5write.c -g -O2 -g -O2 -L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib -L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib -lhdf5 -o tools/h5write
• HDF5_OK=yes
• '[' -z yes ']'
• '[' -z yes ']'
• '[' -z yes ']'
• '[' -z yes ']'
• echo 'Found working hdf5'
  Found working hdf5
• echo 'HDF5_CFLAGS='''''''
  HDF5_CFLAGS=''
• echo 'HDF5_LIBS='''-lhdf5''''
  HDF5_LIBS='-lhdf5'
• CXX17_OK=
• CXX_FS_FLAG=
• printf '\nTesting C++17 filesystem feature support...'

Testing C++17 filesystem feature support...+ /software/gcc/12.2.0/bin/g++ -std=c++17 tools/cxx17_filesystem.cc -g -O2 -g -O2 -L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib -L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib -std=c++17 -o tools/cxx17_filesystem

• CXX17_OK=yes
• '[' -z yes ']'
• '[' -z yes ']'
• '[' -z yes ']'
• printf '\nTesting availability of highway SIMD library...'

Testing availability of highway SIMD library...+ HWY_OK=

• HWY_CFLAGS=-Ibpcells-cpp
• HWY_LIBS=-lhwy
• /software/gcc/12.2.0/bin/g++ -std=c++17 tools/hwy-test.cpp -g -O2 -g -O2 -L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib -L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib
  tools/hwy-test.cpp:1:10: fatal error: hwy/highway.h: No such file or directory
  1 | #include <hwy/highway.h>
  | ^~~~~~~~~~~~~~~
  compilation terminated.
• '[' -z ']'
• '[' '!' -f tools/highway/lib/libhwy.a ']'
• printf '\nBuilding highway SIMD library from source\n'

Building highway SIMD library from source

• CXX='/software/gcc/12.2.0/bin/g++ -std=c++17'
• bash src/vendor/highway/manual-build/build_highway.sh src/vendor/highway tools/highway
• HWY_OK=yes
• HWY_CFLAGS='-Ibpcells-cpp -I../tools/highway/include'
• HWY_LIBS='-L../tools/highway/lib -lhwy'
• '[' -z yes ']'
• sed -e 's|%HDF5_CFLAGS%||g' -e 's|%HDF5_LIBS%|-lhdf5|g' -e 's|%CXX_FS_FLAG%||g' -e 's|%HWY_CFLAGS%|-Ibpcells-cpp -I../tools/highway/include|g' -e 's|%HWY_LIBS%|-L../tools/highway/lib -lhwy|g' -e 's|%ENV_CFLAGS%|-g -O2|g' -e 's|%ENV_LDFLAGS%|-L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib|g' src/Makevars.in
• '[' yes == yes ']'
• curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure-success
  ** libs
  using C++ compiler: ‘g++ (GCC) 12.2.0’
  /software/gcc/12.2.0/bin/g++ -std=c++17 -I"/software/r/4.4.0/lib64/R/include" -DNDEBUG -I'/omics/odcf/analysis/OE0415_projects_temp/nb/pravin_temp/R_temp/Rcpp/include' -I'/omics/odcf/analysis/OE0415_projects_temp/nb/pravin_temp/R_temp/RcppEigen/include' -I/usr/local/include -Ibpcells-cpp -I../tools/highway/include -g -O2 -Ivendor -std=c++17 -DRCPP_EIGEN -DEIGEN_PERMANENTLY_DISABLE_STUPID_WARNINGS -Wno-ignored-attributes -Wno-unknown-pragmas -fpic -g -O2 -c bitpacking_io.cpp -o bitpacking_io.o
  In file included from vendor/highfive/bits/../bits/H5Exception_misc.hpp:14,
  from vendor/highfive/bits/../H5Exception.hpp:164,
  from vendor/highfive/bits/H5Object_misc.hpp:13,
  from vendor/highfive/H5Object.hpp:135,
  from vendor/highfive/H5DataSpace.hpp:17,
  from vendor/highfive/H5DataSet.hpp:13,
  from bpcells-cpp/arrayIO/hdf5.h:13,
  from bitpacking_io.cpp:15:
  vendor/highfive/bits/../bits/h5_wrapper.hpp: In function ‘void HighFive::detail::h5_free_memory(void*)’:
  vendor/highfive/bits/../bits/h5_wrapper.hpp:7:9: error: ‘H5free_memory’ was not declared in this scope; did you mean ‘h5_free_memory’?
  7 | if (H5free_memory(mem) < 0) {
  | ^~~~~~~~~~~~~
  | h5_free_memory
  vendor/highfive/bits/../bits/h5_wrapper.hpp: In function ‘herr_t HighFive::detail::nothrow::h5_free_memory(void*)’:
  vendor/highfive/bits/../bits/h5_wrapper.hpp:14:12: error: ‘H5free_memory’ was not declared in this scope; did you mean ‘h5_free_memory’?
  14 | return H5free_memory(mem);
  | ^~~~~~~~~~~~~
  | h5_free_memory
  make: *** [bitpacking_io.o] Error 1
  ERROR: compilation failed for package ‘BPCells’

• removing ‘/omics/odcf/analysis/OE0415_projects_temp/nb/pravin_temp/R_temp/BPCells’
  Warning messages:
  1: In readLines(old_path) :
  incomplete final line found on '/home/p480n/.R/Makevars'
  2: In i.p(...) :
  installation of package ‘/local/tmp/RtmpRcff7x/file42af440c655d/BPCells_0.2.0.tar.gz’ had non-zero exit status

[bnprks]

Hi @pravinvel98, thanks for your question, and for providing the full debug logs. Apologies on the delay getting back to you.

The error you see in highway compilation is expected -- it is part of testing if you already have a working copy of the library on your system, and if not BPCells simply compiles a working copy from source.

Your main issue seems to be from HDF5, specifically it looks like the version you have enabled is 1.8.12 (based on your environment variable LDFLAGS='-L/tbi/software/x86_64/libhdf5/libhdf5-1.8.12/el7/lib'). The final error is related not finding the function H5free_memory, which the hdf5 documentation says was added in version 1.8.13

The easiest fix would be if you can get a newer version of HDF5 -- 1.8.12 seems to be over 10 years old, and it is evidently not compatible with the hdf5 wrapper library we use (HighFive). Could you try using a newer hdf5 and see if it fixes the H5free_memory error?

(You may need to adjust your LDFLAGS environment variable and maybe add the appropriate include folder to your CFLAGS variable)