Add fch5 an hdf5-based format for storing framecaches #728
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hello @ChristosT! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found: There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻 Comment last updated at 2024-11-13 23:27:13 UTC |
ChristosT
force-pushed
the
add-hdf5-support
branch
5 times, most recently
from
b6dee58 to
4cd1453
Compare
ChristosT
changed the title
psavery
reviewed
Nov 11, 2024
hexrd/imageseries/save.py
Outdated
| h5f["nframes"] = nframes | ||
| h5f["dtype"] = str(self._ims.dtype).encode() | ||
| metadata = h5f.create_group("metadata") | ||
| unwrap_dict_to_h5(metadata, self._process_meta()) |
There was a problem hiding this comment.
Suggested change
| unwrap_dict_to_h5(metadata, self._process_meta()) | |
| unwrap_dict_to_h5(metadata, self._meta) |
Can you just do this instead? I think we don't need self._process_meta() for the fch5 format.
We should also verify that this saves nested metadata correctly!
psavery
reviewed
Nov 11, 2024
Gives access to a plethora of compression algorithms for hdf5 files
.fch5 is an hdf5-based format for saving framecaches comprised mainly 3 datasets - 'data': (m,1) array holding the datavalues of all frames. `m` is evaluated upon runtime - 'indices': (m,2) array holding the row& col information for the values in data. 'data' together within 'indices' represent tha data using the CSR format for sparse matrices. - 'frame_ids`: (2*nframes) holds the range that the i-th frame occupies in the above arrays. i.e. the information of the i-th frame can be accessed using: data_i = data[frame_ids[2*i]:frame_ids[2*i+1]] and indices_i = indices[frame_ids[2*i]:frame_ids[2*i+1]]
ChristosT
force-pushed
the
add-hdf5-support
branch
from
4cd1453 to
a4968ce
Compare
ChristosT
commented
Nov 12, 2024
tests/imageseries/test_formats.py
Outdated
Comment on lines
140
to
149
|
|
||
| def test_fmtfc_nested_metadata(self): | ||
| """frame-cache format with nested metadata""" | ||
| metadata = { 'int': 1, 'array': np.array([1,2,3])} | ||
| self.is_a.metadata["key"] = metadata | ||
|
|
||
| imageseries.write(self.is_a, self.fcfile, self.fmt, style=self.style, | ||
| threshold=self.thresh, cache_file=self.cache_file | ||
| ) | ||
| is_fc = imageseries.open(self.fcfile, self.fmt, style=self.style) | ||
| self.assertTrue(compare_meta(self.is_a, is_fc)) |
There was a problem hiding this comment.
Added test for checking nested metadata support in the new format.
ChristosT
force-pushed
the
add-hdf5-support
branch
from
e16a42b to
6137c15
Compare
ChristosT
force-pushed
the
add-hdf5-support
branch
from
6137c15 to
01e6bbe
Compare
|
This is tested with the gui and works as expected. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The new format allows to store more complex datasets including metadata at multiple levels.
The new fch5 file contains the following fields:
shape,nframes,dtypesimilar to before storing shape as tuple, nframes as integer and dtype of the imageseries as an encoded string.dataan (m,1) array of typedtypeholding the datavalues of all frames.mis evaluated on runtime based onthresholdvalueindicesan (m,2) array of typenp.uint16holding the row and column indices of the datavalues for each frame.frame_ids: (2*nframes,1) holds the range that the i-th frame occupies in the above, i.e. the information of the i-th frame can be accessed using:Both the reader and the writer are multi-threaded and perform better in terms of space efficiency, speed and memory consumption during writes.
Improvements:
Maximum memory usage (as reported by GNU time) saving a raw 7.2G file:
npz format: 18.63 Gb
fch5 format: 1.8 Gb
in terms of space efficiency, npz uses
zip. Forfch5we bring the hdf5plugin package and chose theblosccompression filter withzstdas the underlying compression algorithm and5as the compression level. The choice of the compression configuration was based on trading off between size and speed while being competitive against the currentnpzwriter.This results in the following sizes for the same raw file:
and their corresponding saving time:
Note that similar to the npz case we store datavalues in their native format so they can be either integers or floats.