compatibility with pypy

[mmckerns]

Looks like there are some compatibility issues with dill and pypy: see uqfoundation/pathos#50. Apparently ctypes in pypy does not have a CellType… or at least this import fails:

>>> import ctypes
>>> ctypes.pythonapi.PyCell_New
Traceback (most recent call last): File
"<stdin>", line 1, in <module> AttributeError: 'module' object has no
attribute 'pythonapi'

Initial fix will be to disable the CellType for pypy

[mmckerns]

As noted here: https://pypy.readthedocs.org/en/release-2.4.x/ctypes-implementation.html#discussion-and-limitations, ctypes.pythonapi has been removed from pypy support.

[mmckerns]

Should be fixed in: 07497f7.

Will need some testing for objects that use python's CellType. @imazor, I welcome any help here.

[imazor]

Ok, so I have tested dill with pypy after the fix, (have build and
installed dill from source - from the repository).
Althogut I am not getting the error related to ctypes, there is other error:

Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "run_adjust_etl.py", line 10, in

metadata_fpath='/home/imazor/adjust_etl/high_performance/metadata.pickle')
File "/home/imazor/adjust_etl/high_performance/adjust_logs_parser.py",
line 48, in init
self.curr_metadata, self.parsed_data_json_files =
self.*parse_adjust_events()
File "/home/imazor/adjust_etl/high_performance/adjust_logs_parser.py",
line 428, in __parse_adjust_events
q = manager.Queue()
File "/var/opt/pypy/lib-python/2.7/multiprocessing/managers.py", line
667, in temp
token, exp = self._create(typeid, _args, _kwds)
File "/var/opt/pypy/lib-python/2.7/multiprocessing/managers.py", line
567, in _create
id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds)
File "/var/opt/pypy/lib-python/2.7/multiprocessing/managers.py", line
101, in dispatch
c.send((id, methodname, args, kwds))
File "/var/opt/pypy/lib-python/2.7/pickle.py", line 1417, in dumps
Pickler(file, protocol).dump(obj)
File "/var/opt/pypy/lib-python/2.7/pickle.py", line 224, in dump
self.save(obj)
File "/var/opt/pypy/lib-python/2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/var/opt/pypy/lib-python/2.7/pickle.py", line 562, in save_tuple
save(element)
File "/var/opt/pypy/lib-python/2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File
"/usr/local/lib/python2.7/dist-packages/dill-0.2.3.dev-py2.7.egg/dill/dill.py",
line 849, in save_dictproxy
raise ReferenceError("%s does not reference a class __dict" % obj)
ReferenceError: {} does not reference a class dict

Regarding helping in testing CellType - will be happy, however dont have
much good understand of ctypes (never used it). So actually at the moment
have no clue how to test it.

On Tue, Jan 13, 2015 at 3:27 PM, Mike McKerns notifications@github.com
wrote:

Should be fixed in: 07497f7
07497f7
.

Will need some testing for objects that use python's CellType. @imazor
https://github.com/imazor, I welcome any help here.

—
Reply to this email directly or view it on GitHub
#73 (comment).

--------------------------------------------------------------------------

Igor Mazor
Senior Business Intelligence Manager

Rocket Internet AG | Johannisstraße 20 | 10117 Berlin | Deutschland
skype: igor_rocket_internet | mail: igor.mazor http://goog_862328191
@rocket-internet.de
www.rocket-internet.de

Geschäftsführer: Dr. Johannes Bruder, Arnt Jeschke, Alexander Kudlich
Eingetragen beim Amtsgericht Berlin, HRB 109262 USt-ID DE256469659

[mmckerns]

Ok, so what I'm going to do is just keep removing any problematic objects (hopefully, very few) from the pypy version of dill, and then dill should have a start on pypy with only a very mildly reduced set of coverage. The DictProxyType is close to the end of the file, so hopefully that will be the last error you see on build. Punted on DictProxyType for pypy here: 3e4dd70.

To test what objects build and which don't, once you have an install (built from source)… you'll just need to run one of the tests, and post the output.

Here's the known collection of serialization failures I see for python2.7, and also python3.4 (not pypy):

>$ python27
>$ python test_objects.py 
fails: ListIteratorType <type 'listiterator'>
fails: TupleIteratorType <type 'tupleiterator'>
fails: XRangeIteratorType <type 'rangeiterator'>
fails: CycleType <type 'itertools.cycle'>
fails: PermutationsType <type 'itertools.permutations'>
fails: CombinationsType <type 'itertools.combinations'>
fails: RepeatType <type 'itertools.repeat'>
fails: CompressType <type 'itertools.compress'>

>$ python34
>$ python test_objects.py 
fails: BZ2FileType <class 'bz2.BZ2File'>
fails: BZ2CompressorType <class '_bz2.BZ2Compressor'>
fails: BZ2DecompressorType <class '_bz2.BZ2Decompressor'>
fails: SocketType <class 'socket.socket'>
fails: SocketPairType <class 'socket.socket'>
fails: FileType <class '_io.FileIO'>
fails: GzipFileType <class 'gzip.GzipFile'>

If you could do similarly for your version of pypy, that would be much appreciated.

[imazor]

Ok, so dill with pypy at the moment do give me any error, how ever
pathos.multiprocessing do:

Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "run_adjust_etl.py", line 10, in

metadata_fpath='/home/imazor/adjust_etl/high_performance/metadata.pickle')
File "/home/imazor/adjust_etl/high_performance/adjust_logs_parser.py",
line 48, in init
self.curr_metadata, self.parsed_data_json_files =
self.*parse_adjust_events()
File "/home/imazor/adjust_etl/high_performance/adjust_logs_parser.py",
line 434, in __parse_adjust_events
results = pool.map(universal_worker, func_and_args)
File
"/usr/local/lib/python2.7/dist-packages/pathos-0.2a1.dev-py2.7.egg/pathos/multiprocessing.py",
line 123, in map
return _pool.map(star(f), zip(args)) # chunksize
File "/var/opt/pypy/lib-python/2.7/multiprocessing/pool.py", line 251, in
map
return self.map_async(func, iterable, chunksize).get()
File "/var/opt/pypy/lib-python/2.7/multiprocessing/pool.py", line 558, in
get
raise self._value
AttributeError: 'function' object has no attribute '__objclass'

On Wed, Jan 14, 2015 at 2:55 PM, Mike McKerns notifications@github.com
wrote:

Ok, so what I'm going to do is just keep removing any problematic objects
(hopefully, very few) from the pypy version of dill, and then dill should
have a start on pypy with only a very mildly reduced set of coverage. The
DictProxyType is close to the end of the file, so hopefully that will be
the last error you see on build. Punted on DictProxyType for pypy here:
3e4dd70
3e4dd70
.

To test what objects build and which don't, once you have an install
(built from source)… you'll just need to run one of the tests, and post the
output.

Here's the known collection of serialization failures I see for python2.7,
and also python3.4 (not pypy):

$ python27
$ python test_objects.py
fails: ListIteratorType <type 'listiterator'>
fails: TupleIteratorType <type 'tupleiterator'>
fails: XRangeIteratorType <type 'rangeiterator'>
fails: CycleType <type 'itertools.cycle'>
fails: PermutationsType <type 'itertools.permutations'>
fails: CombinationsType <type 'itertools.combinations'>
fails: RepeatType <type 'itertools.repeat'>
fails: CompressType <type 'itertools.compress'>

$ python34
$ python test_objects.py
fails: BZ2FileType <class 'bz2.BZ2File'>
fails: BZ2CompressorType <class '_bz2.BZ2Compressor'>
fails: BZ2DecompressorType <class '_bz2.BZ2Decompressor'>
fails: SocketType <class 'socket.socket'>
fails: SocketPairType <class 'socket.socket'>
fails: FileType <class '_io.FileIO'>
fails: GzipFileType <class 'gzip.GzipFile'>

If you could do similarly for your version of pypy, that would be much
appreciated.

—
Reply to this email directly or view it on GitHub
#73 (comment).

--------------------------------------------------------------------------

Igor Mazor
Senior Business Intelligence Manager

Rocket Internet AG | Johannisstraße 20 | 10117 Berlin | Deutschland
skype: igor_rocket_internet | mail: igor.mazor http://goog_862328191
@rocket-internet.de
www.rocket-internet.de

Geschäftsführer: Dr. Johannes Bruder, Arnt Jeschke, Alexander Kudlich
Eingetragen beim Amtsgericht Berlin, HRB 109262 USt-ID DE256469659

[mmckerns]

Ok, then I'll close this issue if you ran the dill test that I requested. Should I assume you saw the same output as I posted?

Your last post above is a pathos issue, so I'm going back to the previous ticket.

[imazor]

For dill, I haven't still ran the tests, only my script that was failing.
I will try get to the tests tomorrow.
Am 14.01.2015 18:26 schrieb "Mike McKerns" notifications@github.com:

Ok, then I'll close this issue if you ran the dill test that I requested.
Should I assume you saw the same output as I posted?

Your last post above is a pathos issue, so I'm going back to the previous
ticket.

—
Reply to this email directly or view it on GitHub
#73 (comment).

[mmckerns]

danke

[imazor]

OK, the test results:

pypy --version
Python 2.7.8 (f5dcc2477b97, Sep 18 2014, 11:33:30)
[PyPy 2.4.0 with GCC 4.6.3]

pypy test_objects.py
Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "test_objects.py", line 18, in
load_types(pickleable=True,unpickleable=False)
File
"/var/opt/pypy/site-packages/dill-0.2.3.dev0-py2.7.egg/dill/init.py",
line 60, in load_types
from . import _objects
File
"/var/opt/pypy/site-packages/dill-0.2.3.dev0-py2.7.egg/dill/_objects.py",
line 260, in
a['MemberDescriptorType'] = type.dict['weakrefoffset']
KeyError: 'weakrefoffset'

On Wed, Jan 14, 2015 at 6:31 PM, Mike McKerns notifications@github.com
wrote:

danke sehr

—
Reply to this email directly or view it on GitHub
#73 (comment).

--------------------------------------------------------------------------

Igor Mazor
Senior Business Intelligence Manager

Rocket Internet AG | Johannisstraße 20 | 10117 Berlin | Deutschland
skype: igor_rocket_internet | mail: igor.mazor http://goog_862328191
@rocket-internet.de
www.rocket-internet.de

Geschäftsführer: Dr. Johannes Bruder, Arnt Jeschke, Alexander Kudlich
Eingetragen beim Amtsgericht Berlin, HRB 109262 USt-ID DE256469659

[mmckerns]

Oh… I should have though of that. pypy will have different implementations for certain objects (like the MemberDescriptorType above), so there will be likely be some failures (more than the one you see) in building some of the low-level test objects. It may be that dill can actually serialize these objects in pypy, but the construction of these objects is not the same in pypy as it is in python -- so the test cases will fail to be built, while the objects themselves may not fail to serialize.

The only real way to deal with that is for me (or someone) to work through this test, commenting out each object that fails -- and then later uncommenting them when it's figured out how the object is built in pypy (or if it even exists). I expect there might be 10-20 objects that need attention.

However, if you could run the other tests and let me know which ones fail (no need to paste the traceback -- just the list of who fails), then that will check the major functionality for dill (as opposed to seeing how it serializes particular objects). If not, I'll get to it at some point.

I'm guessing the functionality is such that I can close this ticket, and open new tickets for specific cases where dill has failures in pypy. I've been meaning to work with pypy myself, so maybe this will be a good way to get started.

[imazor]

very well, I have ran all the other tests with pypy.
Except test_temp.py all of them failed.

On Thu, Jan 15, 2015 at 2:52 PM, Mike McKerns notifications@github.com
wrote:

Oh… I should have though of that. pypy will have different
implementations for certain objects (like the MemberDescriptorType
above), so there will be likely be some failures (more than the one you
see) in building some of the low-level test objects. It may be that dill
can actually serialize these objects in pypy, but the construction of
these objects is not the same in pypy as it is in python -- so the test
cases will fail to be built, while the objects themselves may not fail
to serialize.

The only real way to deal with that is for me (or someone) to work through
this test, commenting out each object that fails -- and then later
uncommenting them when it's figured out how the object is built in pypy
(or if it even exists). I expect there might be 10-20 objects that need
attention.

However, if you could run the other tests and let me know which ones fail
(no need to paste the traceback -- just the list of who fails), then that
will check the major functionality for dill (as opposed to seeing how it
serializes particular objects). If not, I'll get to it at some point.

I'm guessing the functionality is such that I can close this ticket, and
open new tickets for specific cases where dill has failures in pypy. I've
been meaning to work with pypy myself, so maybe this will be a good way
to get started.

—
Reply to this email directly or view it on GitHub
#73 (comment).

--------------------------------------------------------------------------

Igor Mazor
Senior Business Intelligence Manager

Rocket Internet AG | Johannisstraße 20 | 10117 Berlin | Deutschland
skype: igor_rocket_internet | mail: igor.mazor http://goog_862328191
@rocket-internet.de
www.rocket-internet.de

Geschäftsführer: Dr. Johannes Bruder, Arnt Jeschke, Alexander Kudlich
Eingetragen beim Amtsgericht Berlin, HRB 109262 USt-ID DE256469659

[mmckerns]

Ach, that's worse than I expected. I'll leave this ticket open, and have to work with it a bit myself. At least dill "as-is" covers some of the basic cases, as the test are for more complex cases. Hopefully that will be good enough to get you going in the meantime… and feel free to keep positing issues you find for pypy compatibility here.

[tarekziade]

I should mention that "AttributeError: 'function' object has no attribute 'objclass'" is not related to multiprocessing. It's a bug happening when you want to pickle a simple function:

$ .tox/pypy/bin/pypy
Python 2.7.10 (f3ad1e1e1d6215e20d34bb65ab85ff9188c9f559, Sep 21 2015, 15:59:29)
[PyPy 2.6.1 with GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> import dill
>>>> def func():
....   pass
....
>>>> dill.dumps(func)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tarek/Dev/github.com/cliquet/.tox/pypy/site-packages/dill/dill.py", line 243, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/tarek/Dev/github.com/cliquet/.tox/pypy/site-packages/dill/dill.py", line 236, in dump
    pik.dump(obj)
  File "/usr/local/Cellar/pypy/2.6.1/libexec/lib-python/2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/local/Cellar/pypy/2.6.1/libexec/lib-python/2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/tarek/Dev/github.com/cliquet/.tox/pypy/site-packages/dill/dill.py", line 1015, in save_wrapper_descriptor
    pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__,
AttributeError: 'function' object has no attribute '__objclass__'

[mmckerns]

There's a patch that was recently submitted (#131), and I don't remember why I didn't merge it b/c I didn't comment on it… I think because I don't have pypy installed. Maybe either of you (@tarekziade or @imazor) can test this PR? basically, apply it, then run the dill test suite and let me know the results? For whatever reason, I have not gotten around to setting up a virtual env with pypy, and testing it myself. Maybe I'm lazy… i don't know. Any help here would be appreciated. I tend to run the dill test suite for at least python2.7, python3.4, and python3.5… but if it's a bigger change, then I do 2.5-3.5.

[Peque]

@mmckerns I would be interested in this issue. I am trying to add support for dill in Pyro4, but tests are failing with pypy. Did you (or @tarekziade or @imazor) manage to try #131? Otherwise I could try to help with this myself, if I can. 😉

[Peque]

@mmckerns also, have you thought about adding a .travis.yml config file to allow automatic testing of pull requests?

[matsjoyce]

@Peque See #3.

[mmckerns]

I've just installed pypy, and I'll see what I can do with this myself now that I can run the tests and see how things fail. On first pass, I was able to reproduce what people have seen above. Next step is to try out the patch.

[mmckerns]

It looks like after 9967dd7 only a few of the tests still fail -- test_*, where *=(classdef, detect, diff, file, objects, weakref), I think. So, I should be able to work on those one by one. I'm not sure if I'll close this and open new tickets for each test failure… that might be the best if any of the tests are hard to resolve.

[mmckerns]

Test failures look minor, as noted here: 044dd1d. Generally seem to be due to *DescriptorType objects. Additionally, one of the file modes fail, and there seems to be some different attributes on the object type between pypy and python that have an impact on the diff module. Since these seem to be minor, I think fixing all in the context of this ticket seems feasible. Errors noted in the above tests with FIXME: pypy.

[Peque]

@mmckerns Any updates on this issue? 😊

[mmckerns]

@Peque: I've been otherwise occupied for about a month. I'll return to this issue and see if I can close it up when I'm no longer on fire -- I expect that will be in a few days.

[EntilZha]

Also encountered this issue for similar reasons from @Peque. I am using dill to provide good pickle support for parallelization in https://github.com/EntilZha/PyFunctional.

[mmckerns]

@EntilZha: interesting module. You may want to also check out my pathos (or multiprocess and ppft), and klepto modules. Also, if you encounter specific issues with pypy compatibility, please feel free to post them here or in their own ticket.

[mmckerns]

All tests for pypy pass as of 1ac1750. The only "exception" is test_file, which is failing due to a bug in pypy (https://bitbucket.org/pypy/pypy/issues/1233). So, I'm closing this issue. New pypy incompatibilities discovered should be added as new tickets. If there are issues from the above that are unresolved, I'll reopen this ticket and deal with them here -- or otherwise as appropriate.

[Peque]

@mmckerns that is really good news, thanks for your hard work! 😊

[mmckerns]

It's probably worth noting that a slightly different set of things pickle under pypy, but I think that's to be expected. I also didn't try any of the other dill.settings, but those should not be different enough to cause issues. I think.

[EntilZha]

Any idea when this would go out to PyPI?

[mmckerns]

You mean, when will the next release be? I'm shooting for less than a month from now.

[Peque]

@mmckerns How do you launch the tests? In my case, I get errors with:

$ pypy tests/test_objects.py 
fails: CCharType <class 'ctypes.c_char'>
fails: CWCharType <class 'ctypes.c_wchar'>
fails: CByteType <class 'ctypes.c_byte'>
fails: CUByteType <class 'ctypes.c_ubyte'>
fails: CShortType <class 'ctypes.c_short'>
fails: CUShortType <class 'ctypes.c_ushort'>
fails: CIntType <class 'ctypes.c_int'>
fails: CUIntType <class 'ctypes.c_uint'>
fails: CLongType <class 'ctypes.c_long'>
fails: CULongType <class 'ctypes.c_ulong'>
fails: CLongLongType <class 'ctypes.c_long'>
fails: CULongLongType <class 'ctypes.c_ulong'>
fails: CFloatType <class 'ctypes.c_float'>
fails: CDoubleType <class 'ctypes.c_double'>
fails: CSizeTType <class 'ctypes.c_ulong'>
fails: BigEndianStructureType <class 'ctypes._endian.BigEndianStructure'>
fails: CSSizeTType <class 'ctypes.c_long'>
fails: DictProxyType <type 'dict'>
fails: DictProxyType2 <type 'dict'>
fails: GetSetDescriptorType <type 'getset_descriptor'>
fails: UnboundMethodType <type 'method'>
fails: ConnectionType <class '_sqlite3.Connection'>
fails: CursorType <class '_sqlite3.Cursor'>
fails: BZ2FileType <type 'BZ2File'>
fails: BZ2CompressorType <type 'BZ2Compressor'>
fails: BZ2DecompressorType <type 'BZ2Decompressor'>
fails: CycleType <type 'itertools.cycle'>
fails: GzipFileType <class 'gzip.GzipFile'>
fails: PermutationsType <type 'itertools.permutations'>
fails: CombinationsType <type 'itertools.combinations'>
fails: RepeatType <type 'itertools.repeat'>
fails: CompressType <type 'itertools.compress'>
fails: SuperType <type 'super'>

And:

$ pypy tests/dill_bugs.py 
True: <type 'str'>, __main__
True: <type 'function'>, <function _method at 0x0000000002534ea8>
Traceback (most recent call last):
  File "tests/dill_bugs.py", line 42, in <module>
    print ("%s: %s, %s" % (dill.pickles(i), type(i), i))
  File "/media/src/dill/dill/dill.py", line 1304, in pickles
    pik = copy(obj, **kwds)
  File "/media/src/dill/dill/dill.py", line 206, in copy
    return loads(dumps(obj, *args, **kwds))
  File "/media/src/dill/dill/dill.py", line 256, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/media/src/dill/dill/dill.py", line 249, in dump
    pik.dump(obj)
  File "/usr/lib64/pypy-4.0.1/lib-python/2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib64/pypy-4.0.1/lib-python/2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/media/src/dill/dill/dill.py", line 1008, in save_wrapper_descriptor
    pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__,
AttributeError: generic property has no __objclass__

Could this AttributeError be related to the errors we are getting when trying to pass Pyro4 tests with dill and pypy?

[mmckerns]

@Peque: I launch them manually. I know, shame on me for not having a better testing infrastructure, but it simply hasn't happened yet. So, I get the same results as you… however, these are expected. Those two particular tests are not typical tests, in the sense that they produce output. The first one, test_objects.py is expected to produce output for all of the objects that fail to serialize -- essentially, it gives a list of what doesn't work. I guess I could edit the test to not produce output by confirming successes and failures against a known list, but currently this is not done. The second test is not even a test (note dill_bugs.py doesn't begin with test_), but merely a reminder of certain important failure cases. I guess it could also be modified to confirm that things are failing as they are currently expected to fail.

In short, if that is the output you are receiving, from all the tests in the test directory, then you are seeing all tests PASS.

[mmckerns]

However, the output from dill_bugs.py is not as is expected… so there is indeed something there. Yes, it seems that it is related to the errors you are getting here: irmen/Pyro4#109 (comment).

I'm going to open the above as a new ticket (#175).

[Peque]

@mmckerns Once again, thanks for your effort and for making dill a better tool. 😊