WIP: Save SpatialImages into HDF5 files.

[alexsavio]

First commit: functions to save a Nifti1Image into a HDF5 file group and retrieve it back.

[matthew-brett]

Thanks for this.

I had a quick look.

If I understand right, you just save the data, affine, extra components as
separate named datasets at the given path, then extract the header as a
dictionary and put that into the 'attrs' of the 'data' dataset.

I guess we lose header.extensions - but that would need special code.

The advantage over MINC2, as I understand it, is that you can store more than
one image in a single HDF5 file.

This idea makes me think again about whether we want our own data format to save
to. It's a similar idea to the stuff that Brendan Moloney is working on with
the SpatialMeta object in his current pull request.

I mean a format that could capture what we need from Nifti, MINC, DICOM and save
it to some sensible format, HDF5 or not - then we could pull any of MINC, Nifti,
even DICOM off that format.

I realize the question's a bit general - I'll try and post to the list with more
thoughts.

[alexsavio]

Hi,

I'm sorry for not replying previously.
I was waiting for further decisions from you.

I'm using the code I uploaded here and had to do one change lately.

If you like it I can develop this idea more.

Thanks!

Kind regards,
Alex

Alexandre Manhães Savio alexsavio@gmail.com
Grupo de Inteligencia Computacional http://www.ehu.es/ccwintco
Departamento de CCIA
UPV/EHU

On 12 December 2013 01:54, Matthew Brett notifications@github.com wrote:

Thanks for this.

I had a quick look.

If I understand right, you just save the data, affine, extra components as
separate named datasets at the given path, then extract the header as a
dictionary and put that into the 'attrs' of the 'data' dataset.

I guess we lose header.extensions - but that would need special code.

The advantage over MINC2, as I understand it, is that you can store more
than
one image in a single HDF5 file.

This idea makes me think again about whether we want our own data format
to save
to. It's a similar idea to the stuff that Brendan Moloney is working on
with
the SpatialMeta object in his current pull request.

I mean a format that could capture what we need from Nifti, MINC, DICOM
and save
it to some sensible format, HDF5 or not - then we could pull any of MINC,
Nifti,
even DICOM off that format.

I realize the question's a bit general - I'll try and post to the list
with more
thoughts.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/215#issuecomment-30380349
.

[matthew-brett]

Thanks for reminding me. Sorry, I got submerged under a sea of Christmas and New Year comestibles.

I will post to the list in a day or two. The reason I'm being slow is because I'm trying to think what our own image format would look like, and that will take some serious work and thinking that I haven't been able to do yet.

[alexsavio]

Hi,

Thanks for your reply!

I was planning to use this as a tool for compression, help in the
management of big datasets and maybe in the future be able to use the HDF
thread safe benefits from distributed running software.

Using nipype lately, I have been thinking on programming an HDFDataSource
for nipype.
Although I've also seen they are trying to formalize Neuroscience data with
standards such as NI-DM (
http://nbviewer.ipython.org/github/ni-/notebooks/blob/master/NIDMIntro.ipynb
).

HDF may be a good support for all this, or not...

Cheers,
Alex

Alexandre Manhães Savio alexsavio@gmail.com
Grupo de Inteligencia Computacional http://www.ehu.es/ccwintco
Departamento de CCIA
UPV/EHU

On 14 January 2014 16:54, Matthew Brett notifications@github.com wrote:

Thanks for reminding me. Sorry, I got submerged under a sea of Christmas
and New Year comestibles.

I will post to the list in a day or two. The reason I'm being slow is
because I'm trying to think what our own image format would look like, and
that will take some serious work and thinking that I haven't been able to
do yet.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/215#issuecomment-32276634
.

[matthew-brett]

Oops - got buried again when traveling. I need to clear my inbox, but I hope to get to this by the beginning of next week.

The NI-DM looks pretty intimidating.

[alexsavio]

No sweat. Thx.

Intimidating, yes indeed.

Alexandre Manhães Savio alexsavio@gmail.com
Grupo de Inteligencia Computacional http://www.ehu.es/ccwintco
Departamento de CCIA
UPV/EHU

On 23 January 2014 22:33, Matthew Brett notifications@github.com wrote:

Oops - got buried again when traveling. I need to clear my inbox, but I
hope to get to this by the beginning of next week.

The NI-DM looks pretty intimidating.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/215#issuecomment-33171047
.

[larsoner]

If someone wants to resurrect this at some point, I wrote some code that allows saving Python objects to HDF5:

https://github.com/LABSN/expyfun/blob/master/expyfun/io/_hdf5.py

It can handle lists, strings, floats, ints, ndarrays, dicts, etc. and nesting of these types. There are some minor restrictions right now (e.g., all dict keys must be strings), but it might nonetheless be useful.

[bcipolli]

I looked this over, and this seems more useful for defining datasets, rather than individual images. I think of nibabel as working on individual images, not datasets. My thought is to close this...

Perhaps something useful would be to have a website example of how to use HDF5 to store an entire dataset in a single HDF5 file (which is what the pymvpa folks did for the Haxby 2001 and 2011 datasets).

[matthew-brett]

Ben - can you give links to the PyMVPA hdf5 datasets?

[bcipolli]

@matthew-brett please use http://data.pymvpa.org/datasets/hyperalignment_tutorial_data/hyperalignment_tutorial_data_2.4.hdf5.gz.

If you'd like code for downloading and loading the file, please install nidata and run:
ipython nidata/atlas/haxby_etal_2011/example1.py