Support handling faults with GIL released #65
Labels
Comments
|
Another (horrid) approach would be checking is value starts on a page boundary, and if so, loop touching first byte of every page |
Closed
dw
added a commit
that referenced
this issue
Jun 6, 2015
This seems to be the simplest, lowest overhead and most portable solution to the GIL page fault problem: simply loop over the MDB_val, touching one byte every 4kb. Regardless of whether the value lives in a leaf page or an overflow page, is 100 bytes or 64kb, this should ensure on all but the most heavily overloaded machines that the pages are in RAM before PyString_FromStringAndSize() attempts to copy them.
dw
added a commit
that referenced
this issue
Jun 6, 2015
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently when buffers=False, calling
mdb_get(),mdb_cursor_get()and suchlike, the GIL is held while in order to make use of obmalloc by way ofPyString_FromStringAndSize().This means it is currently impossible to use a Python thread to handle a page fault that does not block the main thread, which e.g. could be running an asynchronous IO loop. That situation is particularly likely when copying an oversized value, where
mdb_get()might have faulted the leaf page, but did not fault all the pages dedicated to the oversize value.There are several ways this could be fixed, perhaps by allocating a reasonably sized string before dropping the lock, or similar
The text was updated successfully, but these errors were encountered: