Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

skip omegaconf to avoid stack overflow due to recursive calls #379

Closed
wants to merge 2 commits into from
Closed

skip omegaconf to avoid stack overflow due to recursive calls #379

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3560158
skip omegaconf to avoid stack overflow due to recursive calls
fpzh2011 Nov 29, 2023
a4b7ca2
Merge branch 'main' into skip_omegaconf
jackalcooper Nov 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/onediff/infer_compiler/transform/manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ def update_class_proxies(self, class_proxy_dict: Dict[str, type], verbose=True):
self.logger.debug(debug_message)

def _transform_entity(self, entity):
if "omegaconf" in str(entity):
return entity
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个写得太具体了，没法合并。需要理清为什么，然后想个更系统的策略。

是因为那个 getattr 带来的吧

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个之前跟冯文反复讨论过。暂时只想到这种方式来处理。

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里为什么会导致递归调用呢，把原因写清楚吧。

即使要这样，也需要十足的理由。

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我在这个分支简化了这块的工作逻辑，#376

可以尝试下看有没有关系

Copy link
Contributor

@ccssu ccssu Dec 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

omegaconf 不赖torch 。mock中应该直接是返回 omegaconf 包中对象。这个pr 应该不用修改。
现在这个导致递归调用 是现在mock 机制的问题 . 具体的稍后 我找个 demo 。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#399

我这里注册之后，应该不会触发这个问题了。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

下面代码可复现导致递归的问题。

from omegaconf.dictconfig import DictConfig
from onediff.infer_compiler.transform import register, torch2oflow

config = DictConfig({"local_server": {"port": 8080, "host": "localhost"}})

# ### 解决方式 1
# def func(mod: DictConfig):
#     return mod
# register(torch2oflow_funcs=[func])
# ### end 解决方式 1

of_config = torch2oflow(config)
输出 
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'omegaconf.base.ContainerMetadata'> e=RecursionError('maximum recursion depth exceeded while calling a Python object')
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'omegaconf.dictconfig.DictConfig'> e=RecursionError('maximum recursion depth exceeded')
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'omegaconf.nodes.AnyNode'> e=RecursionError('maximum recursion depth exceeded')
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'omegaconf.nodes.AnyNode'> e=RecursionError('maximum recursion depth exceeded')
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'type'> e=AttributeError("module 'builtins' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'type'> e=AttributeError("module 'builtins' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'type'> e=AttributeError("module 'builtins' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'type'> e=AttributeError("module 'builtins' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'type'> e=AttributeError("module 'builtins' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'type'> e=AttributeError("module 'builtins' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'type'> e=AttributeError("module 'builtins' has no attribute '__path__'")
ERROR [2023-12-05 08:25:47] - Unsupported type: <class 'typing._SpecialForm'> e=AttributeError("module 'typing' has no attribute '__path__'")

result = self.mocker.mock_entity(entity)
if result is None:
RuntimeError(f"Failed to transform entity: {entity}")
Expand Down