Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support different cleaner strategies #28

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jk8172
Copy link

@jk8172 jk8172 commented May 10, 2017

Refactor to support the direct closing of the MappedByteBuffers and to solve issue #27.
The closing/cleanup is moved out of the ReadOnlyMemMap into separate cleaner classes.

The current behaviour with a single threaded executor, which is used to schedule the cleanup with a 1 second delay, is implemented in SingleThreadedCleanerWithWait and kept as the default.

But this default can lead to a severe issue (#27), if we create many readers very fast.
Even if we close them directly after we created them, the memory will be freed only after 1 second at the earliest. Even worse is the fact that we can create readers much faster (with multiple threads) than the single threaded executor can do the cleanup. With high enough rates we can use up the virtual memory space, resulting in an OutOfMemory exception.

As we do not share the (not thread safe) readers between threads, we want the cleanup to happen as soon as we close the reader. So the memory is freed as soon as possible (and the caller thread is blocked).
This is implemented with the DirectCleaner.

To use the DirectCleaner instead of the default behaviour, you have to set a DirectCleaner instance as the cleaner to be used by ReadOnlyMemMap.
This can (currently) only be done globally and not per reader via the setMappedByteBufferCleaner method.

Refactor to support the direct closing of the MappedByteBuffers.
The closing/cleanup is moved out of the ReadOnlyMemMap into separate
cleaner classes.

The current behaviour with a single threaded executor, which is used to
schedule the cleanup with a 1 second delay, is implemented in SingleThreadedCleanerWithWait.
This one is kept as the default.

This default lead to a severe issue, if we create many readers very fast.
Even if we close them directly after we created them, the memory will be
freed only after 1 second at the earliest. Even worse is the fact that we can
create readers much faster (with multiple threads) than the single
threaded executor can do the cleanup. With high enough rates we can
use up the virtual memory space, resulting in an OutOfMemory exception.

As we do not share the (not thread safe) readers between threads, we want
the cleanup to happen as soon as we close the reader. So that the memory
is freed as soon as possible (and the caller thread is blocked).
This is implemented with the DirectCleaner.

To use the DirectCleaner instead of the default behaviour, you have to set
a DirectCleaner instance as the cleaner to be used by ReadOnlyMemMap.
This can (currently) only be done globally and not per reader via the
setMappedByteBufferCleaner method.
@krka
Copy link
Contributor

krka commented Jun 2, 2017

Sorry for the long delay. I am on parental leave and not paying too much attention to work at the moment.

This change seems reasonable, but I would greatly prefer it the settings were per object instead of being static. Otherwise you would get unexpected effects if two separate usages of sparkey tried to use different cleaner settings.

@spkrka
Copy link
Member

spkrka commented Feb 19, 2019

Sorry for waking an old PR... I think version 2.3.2 solves this for you.
From the changelog:

  • Running close() on a SparkeyReader will now synchronously unmap the files.
    This may block for 100 ms if the reader has been duplicated (typically when used from multiple threads).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants