Installation issues with support for Python API

[ritiek]

@ax3l I am really glad you're taking much efforts to have people adopt these standards which I hope people really do!

Install from source must be as easy as cmake .. && make && make install, otherwise we failed.

Yep, it did build smoothly with no problems when I passed no arguments but there was some trouble when passing -DopenPMD_USE_PYTHON=ON and -DopenPMD_USE_HDF5 to cmake.

It's been a while since I tried to build but AFAIK there was some error regarding MPI and HDF5 which I think building HDF5 and h5py with MPI support fixed it. I think it would be very nice if it were point to relevant docs for building with MPI support. (Also, it isn't immediately clear what MPI support does, it might be good to have some docs what additional features it brings to OpenPMD-API :)

That error regarding HDF5 and MPI went away but then now I face this one.

-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Can NOT find 'adios_config' - set ADIOS_ROOT, ADIOS_DIR or INSTALL_PREFIX, or check your PATH
-- Could NOT find ADIOS (missing: ADIOS_LIBRARIES ADIOS_INCLUDE_DIRS) (Required is at least version "1.13.1")
CMake Error at CMakeLists.txt:176 (find_package):
  Could not find a package configuration file provided by "pybind11"
  (requested version 2.2.1) with any of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

  Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
  "pybind11_DIR" to a directory containing one of the above files.  If
  "pybind11" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!
See also "/home/ritiek/Downloads/openPMD-api-build/CMakeFiles/CMakeOutput.log".

I tried looking up and I think installing ADIOS might fix it but it was mentioned in under optional I/O dependencies in the README (and/or it could be that I have pybind installed incorrectly) but I didn't really try installing it yet. I guess I'll tinker a bit more and might raise an issue in OpenPMD-API repo with feedback on what could have been better in my perspective. :)

---

Also, I tried installing it with spack

spack install openpmd-api +python
spack load --dependencies openpmd-api

but for some reason spack always ended up installing Python 2.7.15 even when I created a symbolic link in /usr/bin/python which pointed to my Python 3.6 binary (I wanted it build OpenPMD-API for Python 3.6).

Initially it did say it was running Python 3.6 with spack python but that changed when I did spack install openpmd-api +python. I am not really sure whether this is something to do with OpenPMD-API or spack itself. I haven't used spack before this so I can't say how it works. Do you guys know of a way to specify the Python version?

And conda installation really did go smooth, no errors in da way. :D

---

That said, I am myself very new to HDF5 and OpenPMD in general, so it may be just something to do with my lack of clear understanding at the moment. :/

---

This report is just a copy-paste from PlasmaPy/PlasmaPy-PLEPs#13 (comment).

[ax3l]

Dear @ritiek, thank you a lot for trying out the API and your detailed report!

there was some trouble when passing -DopenPMD_USE_PYTHON=ON

• Yes, the trouble here came from the missing dependency pybind11. Usually, one would install it in advance. Anyway, we will now remove this inconvenience now by shipping it with openPMD-api: Ship pybind11 v2.2.3 #281

it isn't immediately clear what MPI support does

• huh, good point. Indeed, Enable MPI support is a too short CMake string and our official docs could add a section with more intro to it: Doc: MPI #449

MPI support for openPMD-api means: MPI parallel backend support for clusters. For example, parallel HDF5 or MPI aggregated ADIOS1 writing. MPI-enabled libraries should always be build from source on clusters, for various reasons (stability, hardware support such as infiniband, performance).

installing ADIOS might fix it

Nope, adios is just yet another backend that can write serial and MPI parallel files :-) You want ADIOS if you write on large clusters with more than ~1'000 MPI ranks (for example, see: https://arxiv.org/abs/1706.00522).

for some reason spack always ended up installing Python 2.7.15

• Argh, yes sorry for that!
  I improved the documentation in Spack: Python3 #278

And conda installation really did go smooth, no errors in da way. :D

yay, excellent! :)

I am myself very new to HDF5 and OpenPMD in general

Don't worry, these are exactly the reports we like :-)

For further information and background, have you seen our README.md and manual yet? :-)

[ax3l]

@ritiek just for curiosity, can you try to compile the latest dev again from source and see if this works for you as well?

[ritiek]

I just tried to build it from source and it seems to build fine now! I'll also try again with spack soon and let you know how it goes.

[ax3l]

Jippey, thank you!

[ritiek]

Spack installs python 3.6.5 too!

Jippey, thank you!

I should be the one to thank you. So, thank you! :D

[ax3l]

Hi @ritiek, the latest release 0.5.0a fixes a couple of python related issues, mainly attribute writing and proper float handling in datasets. Make sure to update in case you got any problems trying it! :)

[ritiek]

That's great to know @ax3l! Though, there hasn't been much stuff going on around OpenPMD and HDF5 datasets in PlasmaPy at the moment but we'll keep the newer OpenPMD-api version in mind for the future. :)

[ax3l]

Alright, sounds good! Thanks for the feedback :)