Python (Numpy) API

[ax3l]

This issue is about providing a general Python API on top of C++ that can be used in various projects, both parallel and serial.

Possible libraries for bindings:

• https://github.com/pybind/pybind11
• xtensor-python

Ideally, it would be compatible to libs such as Dask #12 and shall very numpy-like for low-level data-access.

[CFGrote]

Hi @ax3l,
i was just about to start sth. similar in simex when i saw this.
i can help out here if needed. any pointers how to get started (prototype, specification etc?)

[ax3l]

Hi @CFGrote , great to read you!
We are short before a first useful prototype, sorry I did not ping you yet!

Shall we quickly give each other a call? I will be leaving for a working travel soon and am a bit in a hurry these days...

[ax3l]

Please also see #59 :) It's quite far :)

[CFGrote]

hi @ax3l, no rush, i can wait for the prototype. have to clear my desk first, anyway. let's touch base when you're back from travel or otherwise find the time. carsten

…

On 02/26/2018 05:17 PM, Axel Huebl wrote: Hi @CFGrote <https://github.com/cfgrote> , great to read you! We are short before a first useful prototype, sorry I did not ping you yet! Shall we quickly give each other a call? I will be leaving for a working travel soon and am a bit in a hurry these days... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOwoRzdKj6rygw2UqafEFoVZvR_-xn7Iks5tYtkUgaJpZM4SASzI>.

-- Dr. Carsten Fortmann-Grote Scientist for Scientific Simulations / Wissenschaftler fuer wissenschaftliche Simulationen European XFEL GmbH Holzkoppel 4 22869 Schenefeld Germany Phone: +49 (0)40 8998-5603 Fax: +49 (0)40 8998-1905 Email: carsten.grote@xfel.eu Web: www.xfel.eu Managing Directors: Prof. Dr. Robert Feidenhans'l Registered as European X-Ray Free-Electron Laser Facility GmbH at Amtsgericht Hamburg, HRB 111165

[ax3l]

Current status of the Python bindings

As soon as #59 is merged (now), the initial structure, a mini-example, compile infrastructure with CMake and dependencies and CI is available. One can then start to iterate over attributes and the common openPMD objects.

The next steps are

• skip for now: add a setup.py (source wheel): this is actually just a CMake wrapper and can be really hard for users to understand/control, especially with the transitive C dependencies (HDF5, ADIOS); (binary wheel): tricky -> I would skip this for now, because to make it useful one would need to pack too many C-source-dependencies in it which is hard to understand if things go wrong for python users setup.py for sdist #240
• add actual data reading into ndarrays:
  • the python buffer protocol can be used for this: https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html; as an example, this is how ADIOS2 does it for their native bindings First Python Numpy Bindings #219
  • an orthogonal way of coupling directly into numpy is by adding high-level xtensor objects for the actual data and binding those. this can also be done as a refactoring later on and should be coordinated with me for dependency management
• python unit tests: I currently prefer nosetests in pure python projects; here we probably just add tests via the add_test examples in CMakeLists.txt (I will have at least one example in there when Python Bindings: Initial Structure #59 is merged), Carsten's unit test draft
• mpi4py (MPI) support: Python MPI Support #99 mpi4py #454

@CFGrote as spoken on the phone: if you are interested in reading into the numpy docs, checking the object lifetimes, etc. for the protocol buffer / numpy bindings and providing PRs that bind those for the actual read of particle and mesh records this would be a wonderful contribution :) I think you can start as soon as #59 is in.

Contribution Guidelines

P.S. If anything is missing in your sphinx manual that you need for development, do not hesitate to open issues or PRs! The sources lie in docs/source/.

[ax3l]

@CFGrote first bindings and an example are now merged with #59 :)
Feel free to start you numpy (python buffer protocol) experiments when you have the time! :)

[ax3l]

@CFGrote we fixed a couple of bugs in the last weeks and I was wondering if you have time to continue on the buffer protocol these days? :)

[CFGrote]

hi axel, thanks. time is extremely sparse these days. let's talk (plus maybe do some work) during the EUCALL AM, next week. c.

…

----- Original Message ----- From: "Axel Huebl" <notifications@github.com> To: "openPMD/openPMD-api" <openPMD-api@noreply.github.com> Cc: "Carsten Fortmann-Grote" <carsten.grote@xfel.eu>, "Mention" <mention@noreply.github.com> Sent: Tuesday, 15 May, 2018 17:19:38 Subject: Re: [openPMD/openPMD-api] Python (Numpy) API (#32) @CFGrote we fixed a couple of bugs in the last weeks and I was wondering if you have time to continue on the buffer protocol these days? :)

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #32 (comment) -- Regards, Carsten Dr. Carsten Fortmann-Grote Scientist for Scientific Simulations / Wissenschaftler fuer wissenschaftliche Simulationen European XFEL GmbH Holzkoppel 4 22869 Schenefeld Germany Phone: +49 (0)40 8998-5603 Fax: +49 (0)40 8998-1905 Email: carsten.grote@xfel.eu Web: www.xfel.eu Managing Directors: Prof. Dr. Robert Feidenhans'l Registered as European X-Ray Free-Electron Laser Facility GmbH at Amtsgericht Hamburg, HRB 111165

[ax3l]

Minimalistic draft of a non-slice, C++ pass-through store_chunk method: https://github.com/ax3l/openPMD-api/tree/topic-storeChunk (see commit message for details)

[ax3l]

@CFGrote I opened a work-in-progress for numpy bindings in #219 now.

Please feel free to share your work on the python unit tests in a separate PR so we can iterate on it.

[ax3l]

I added a first numpy draft in #219 now.

Open are the following points for which I really need unit tests to develop them further.

To Do

• add unit tests unittests for python API #249
• ship pybind11 in share/openPMD/thirdParty/ Ship pybind11 v2.2.3 #281
• add all C++ data types we support
• check numpy data types "round-trip" in writes and reads
• expose attributes Python: Expose Attributes Fully #266
• store_chunk: implement all drafted checks
• load_chunk: cross check allocation only once & move (minimal possible footprint) Not clear how to expose existing C++ vector as numpy array pybind/pybind11#1042 (comment)
• store_chunk/load_chunk: py::arrays: block with numpy flags until flushed
• store_chunk/load_chunk: for offsets and extents just allow simple python arrays Python: Expose Attributes Fully #266
• store_chunk/load_chunk: slice protocol load/storeChunk: slice view #138
• record_component must look like a simplified numpy array: proper types of .dtype, .shape, access offset/extent with int numpy arrays as well, etc.
• add dependency: numpy
• add a gazillion doc strings to the methods and classes (NumPy Style! Sphinx compatible!)

[ax3l]

Pretty widely implemented and usable these days.
Doc-strings still very welcome as PRs! :)