Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursive self-references that trigger a RecursionError #662

Open
cdzhan opened this issue Jun 13, 2024 · 0 comments
Open

Recursive self-references that trigger a RecursionError #662

cdzhan opened this issue Jun 13, 2024 · 0 comments

Comments

@cdzhan
Copy link

cdzhan commented Jun 13, 2024

Error Message

(pt) fooooo@bjys1095:/projs/framework/fooooo/test$ python test_dump.py 
/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py:414: PicklingWarning: Cannot locate reference to <class '__main__.test_dump.<locals>.MyFunc'>.
  StockPickler.save(self, obj, save_persistent_id)
/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py:414: PicklingWarning: Cannot pickle <class '__main__.test_dump.<locals>.MyFunc'>: __main__.test_dump.<locals>.MyFunc has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)
Traceback (most recent call last):
  File "/projs/framework/fooooo/test/test_dump.py", line 17, in <module>
    test_dump()
  File "/projs/framework/fooooo/test/test_dump.py", line 14, in test_dump
    dumps(a)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 280, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 252, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 420, in dump
    StockPickler.dump(self, obj)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 487, in dump
    self.save(obj)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 1832, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 1098, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 1832, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 1098, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/projs/framework/fooooo/venv/pt/lib/python3.10/site-packages/dill/_dill.py", line 1869, in save_type
    StockPickler.save_global(pickler, obj, name=obj_name)
  File "/usr/local/python3.10/lib/python3.10/pickle.py", line 1071, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <class '__main__.test_dump.<locals>.MyFunc'>: it's not found as __main__.test_dump.<locals>.MyFunc

How To Reproduce

import torch
from dill import dumps

class MyTest:
  def forward(self,):
    class MyFunc(torch.autograd.Function):
      pass
   
def test_dump():
  class MyFunc(torch.autograd.Function):
    pass
  a = MyFunc
  dumps(a)

if __name__ == '__main__':
  test_dump()    # fail
  #dumps(MyTest)    # success, why this case pass?

My Env

dill == 0.3.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant