You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently when buffers=False, calling mdb_get(), mdb_cursor_get() and suchlike, the GIL is held while in order to make use of obmalloc by way of PyString_FromStringAndSize().
This means it is currently impossible to use a Python thread to handle a page fault that does not block the main thread, which e.g. could be running an asynchronous IO loop. That situation is particularly likely when copying an oversized value, where mdb_get() might have faulted the leaf page, but did not fault all the pages dedicated to the oversize value.
There are several ways this could be fixed, perhaps by allocating a reasonably sized string before dropping the lock, or similar
The text was updated successfully, but these errors were encountered:
This seems to be the simplest, lowest overhead and most portable
solution to the GIL page fault problem: simply loop over the MDB_val,
touching one byte every 4kb. Regardless of whether the value lives in a
leaf page or an overflow page, is 100 bytes or 64kb, this should ensure
on all but the most heavily overloaded machines that the pages are in
RAM before PyString_FromStringAndSize() attempts to copy them.
Currently when buffers=False, calling
mdb_get()
,mdb_cursor_get()
and suchlike, the GIL is held while in order to make use of obmalloc by way ofPyString_FromStringAndSize()
.This means it is currently impossible to use a Python thread to handle a page fault that does not block the main thread, which e.g. could be running an asynchronous IO loop. That situation is particularly likely when copying an oversized value, where
mdb_get()
might have faulted the leaf page, but did not fault all the pages dedicated to the oversize value.There are several ways this could be fixed, perhaps by allocating a reasonably sized string before dropping the lock, or similar
The text was updated successfully, but these errors were encountered: