NumPy file formats

[ilfreddy]

It would be a good idea to follow @robertodr 's suggestion and implement writing files using NumPy formats. Currently we write trees only as binary files which limits their reusability to within a given architecture/compiler and as I lately notice, even to a given build.

[stigrj]

I'm sure there's a logical reason for the crash, I can't imagine that the binary should change between builds, even with different compilers.

That said, I agree that a portable data format would be good, but we should coordinate this with the MADNESS group. There was a discussion in Berlin about this last summer, and Harrison wanted a common data format that allowed for data transfer between MADNESS and MRChem. The format we discussed then was HDF5, but I don't know if they have moved forward with this yet.

[robertodr]

HDF5 is extremely flexible, but it's an extremely annoying dependencies to satisfy. See also here: https://cyrille.rossant.net/moving-away-hdf5/

[stigrj]

I will contact MADNESS and see if they have done anything on HDF5. If not we might be able to push through the NumPy format instead.

[ilfreddy]

On 21/11/18 22:06 , Stig Rune Jensen wrote: I will contact MADNESS and see if they have done anything on HDF5. If not we might be able to push through the NumPy format instead. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#157 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADoaJgb4zMRkWNYx2t41ulMdCwtQZP1Xks5uxcBTgaJpZM4YtghT>.

They are definitely a Python-friendly community, so it should not be hard to convince them to drop HDF5 and go for NumPy. Luca

[robertodr]

Let me reiterate that for structured datasets NumPy format is probably not a good idea, but it's certainly quicker to set up than HDF5.