Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support handling faults with GIL released #65

Closed
dw opened this issue May 12, 2014 · 1 comment
Closed

Support handling faults with GIL released #65

dw opened this issue May 12, 2014 · 1 comment

Comments

@dw
Copy link
Collaborator

dw commented May 12, 2014

Currently when buffers=False, calling mdb_get(), mdb_cursor_get() and suchlike, the GIL is held while in order to make use of obmalloc by way of PyString_FromStringAndSize().

This means it is currently impossible to use a Python thread to handle a page fault that does not block the main thread, which e.g. could be running an asynchronous IO loop. That situation is particularly likely when copying an oversized value, where mdb_get() might have faulted the leaf page, but did not fault all the pages dedicated to the oversize value.

There are several ways this could be fixed, perhaps by allocating a reasonably sized string before dropping the lock, or similar

@dw dw added the enhancement label May 12, 2014
@dw
Copy link
Collaborator Author

dw commented May 15, 2014

Another (horrid) approach would be checking is value starts on a page boundary, and if so, loop touching first byte of every page

@dw dw mentioned this issue Apr 26, 2015
dw added a commit that referenced this issue Jun 6, 2015
This seems to be the simplest, lowest overhead and most portable
solution to the GIL page fault problem: simply loop over the MDB_val,
touching one byte every 4kb. Regardless of whether the value lives in a
leaf page or an overflow page, is 100 bytes or 64kb, this should ensure
on all but the most heavily overloaded machines that the pages are in
RAM before PyString_FromStringAndSize() attempts to copy them.
@dw dw closed this as completed Jun 6, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant