Improve the performance of create_virtual_dataset for h5py 3 #232
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using VirtualLayout and VirtualSource directly is slow for a few reasons. The main reason is that VirtualSource does a deepcopy of itself every time you slice it. The high-level code also calls the internal helper _convert_space_for_key(), which is unnecessary for our given slices (it only does anything for unbounded slices), and recreates a selection object each time, which we can do just once. Any further performance improvements here would have to come from speeding up the h5py select() function, which is where most of the time is now spent.
|
Thanks, this looks much better. On my machine the example from #226 now runs in 1.81s instead of 2.7s. |
|
So now the latest version of h5py is causing the tests to segfault. I'm not sure what the h5py 3.6 failure is about. I'm not able to reproduce it locally. |
|
What's the error you are seeing? We noticed that there's a bug in h5py version 3.7 with fillvalue handling which is fixed by this PR: h5py/h5py#2111 |
|
The errors are on the CI builds here. h5py 3.6 is giving this error (which I cannot reproduce locally): h5py 3.7 is segfaulting (which I can reproduce) |
|
Looks like that might be the same issue but I'll check to be sure. |
|
Hi @asmeurer: did you get to the bottom of those segfaults? |
|
I think the issues are upstream h5py problems, so we should be OK to release. I wasn't able to take a look at them. |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using VirtualLayout and VirtualSource directly is slow for a few reasons. The
main reason is that VirtualSource does a deepcopy of itself every time you
slice it. The high-level code also calls the internal helper
_convert_space_for_key(), which is unnecessary for our given slices (it only
does anything for unbounded slices), and recreates a selection object each
time, which we can do just once.
Any further performance improvements here would have to come from speeding up
the h5py select() function, which is where most of the time is now spent. I have not yet investigated if there are any obvious gains that can be made here.
I am still investigating if any of these improvements can be upstreamed to h5py itself so that these workarounds aren't necessary and we can just use the normal high-level APIs.
This takes the example in #226 from 3.66 seconds to 2.77 seconds on my machine. In h5py 2.10, the example takes 2.52 seconds via the old optimizations.