-
Notifications
You must be signed in to change notification settings - Fork 174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve compatibility with "nogil" Python and 3.11 #470
Conversation
This makes a number of changes to improve compatibility with the "nogil" Python fork as well as the upcoming 3.11 release. - Fix _code_reduce for 3.11b0 and nogil Python - Use instr.argval in _walk_global_ops. This avoids adding a special case for 3.11+ (and is useful for nogil Python). In 3.11+, the argval for LOAD_GLOBAL would need to be divided by two to access the correct name. The 'argval' field already stores the correct name. - Set '__builtins__' before constructing de-pickled functions. (Useful for nogil Python) - Fix test_recursion_during_pickling in Python 3.11+. Objects now have a default `__getstate__` method so `__getattr__` was never called, but `__getattribute__` would still be called.
There's still a bug in Python 3.11b0 that prevents the tests from passing: But once that is fixed all the tests should pass. (I've run the tests locally with a patched version of 3.11b0). Also, I've included both changes for "nogil" and CPython 3.11 because some of the changes overlapped. Let me know if you'd prefer me to split up the PRs. |
Thanks a lot for your work on nogil Python -- I'm a big fan of this work! Last time I tried it out, I was running into CloudPickle not working so it is really great to see you are fixing it! |
Codecov Report
@@ Coverage Diff @@
## master #470 +/- ##
==========================================
+ Coverage 84.47% 92.60% +8.13%
==========================================
Files 4 4
Lines 715 717 +2
Branches 157 158 +1
==========================================
+ Hits 604 664 +60
+ Misses 87 32 -55
+ Partials 24 21 -3
Continue to review full report at Codecov.
|
Since we now have a I tested locally with my up-to-date nogil venv and all cloudpickle tests pass though. |
The dask distributed failure is unrelated and will be fixed concurrently in #471. |
@pierreglaser any opinion on the changes in this PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The failures observed on the dowstream distributed CI run are the same as those observed in the master
branch (see for instance the last build in #471).
So +1 for merging this on my side. But before making a release I would like to better understand what's going on with distributed, irrespective of the changes in this PR.
I can reproduce the same 19 failures for distributed Let me merge this and release 2.1.0. |
This makes a number of changes to improve compatibility with the
"nogil" Python fork as well as the Python 3.11+.
Fix
_code_reduce
for 3.11b0 and nogil PythonUse instr.argval in _walk_global_ops. This avoids adding a special
case for 3.11+ (and is useful for nogil Python). In 3.11+, the argval
for LOAD_GLOBAL would need to be divided by two to access the correct
name. The 'argval' field already stores the correct name.
Set
__builtins__
before constructing de-pickled functions. (Usefulfor nogil Python)
Fix test_recursion_during_pickling in Python 3.11+. Objects now have
a default
__getstate__
method so__getattr__
was never called,but
__getattribute__
would still be called.