SharedMemory and ndarray serialization
#1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ctrueden
reviewed
Jul 11, 2024
There was a problem hiding this comment.
Thank you, this looks great!
I have one question for discussion before we merge this: on the Java side in apposed/appose-java#5, the analogous class is named just NDArray rather than ShmNDArray. Should we be consistent with the name between the two Appose implementations? Maybe the Java class should be called ShmNDArray as well?
For consistency with appose-java.
ctrueden
force-pushed
the
schmarrn
branch
8 times, most recently
from
4aba23d
to
80c5476
Compare
By putting shm last, we can give it a default value of None, so that the SharedMemory can be created on the fly if not given. See also apposed/appose-java@4346036
Let's not shadow the built-in type function.
Not likely to make an actual performance difference, but may as well. This also better matches the JSON-encoding behavior of appose-java.
So that these classes are in the same package (the base one) as the Java codebase's equivalents. And so that people don't need to know to import SharedMemory from multiprocessing.shared_memory, which is kind of a keyboardful.
To keep things manageable, we never want Python worker processes cleaning up shared memory blocks. We leave it to the service process to do that, always.
To do this, we subclass Python's SharedMemory class, rather than simply passing it along verbatim anymore. And we implement the __exit__ method to call the Appose SharedMemory subclass's dispose() method, which calls either close() or unlink() depending on whether the unlink_on_dispose flag is set. Note that this implementation cannot quite align with the Java one, because in Java, the AutoCloseable interface always calls close(). As such, the close() method must be overridden and taught to sometimes call unlink(), and sometimes not, depending on the unlinkOnClose flag.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a class
ShmNDArraythat annotates SharedMemory with dtype and shape.If NumPy is available
ShmNDArray.ndarray()provides anumpy.ndarrayview.See this usage example in the corresponding apposed/appose-java#5 PR.