Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RecursionError: maximum recursion depth exceeded while linting a large chained method calls #8842

Closed
yilei opened this issue Jul 11, 2023 · 6 comments · Fixed by pylint-dev/astroid#2385 or pylint-dev/astroid#2390
Closed

RecursionError: maximum recursion depth exceeded while linting a large chained method calls #8842

yilei opened this issue Jul 11, 2023 · 6 comments · Fixed by pylint-dev/astroid#2385 or pylint-dev/astroid#2390
Labels
Crash 💥 A bug that makes pylint crash
Milestone
3.1.0

Comments

@yilei
Copy link
Contributor

yilei commented Jul 11, 2023

Bug description

When parsing the following file:

from a import b

(
    b.builder('name')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .add('name', value='value')
    .Build()
)

Configuration

No response

Command used

pylint t.py

Pylint output

Traceback (most recent call last):
  File "pylint/lint/pylinter.py", line 1044, in get_ast
    return MANAGER.ast_from_file(filepath, modname, source=True)
  File "astroid/manager.py", line 124, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
  File "astroid/builder.py", line 145, in file_build
    return self._post_build(module, builder, encoding)
  File "astroid/builder.py", line 173, in _post_build
    module = self._manager.visit_transforms(module)
  File "astroid/manager.py", line 95, in visit_transforms
    return self._transform.visit(node)
  File "astroid/transforms.py", line 89, in visit
    return self._visit(module)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 61, in _visit_generic
    return [self._visit_generic(child) for child in node]
  File "astroid/transforms.py", line 61, in <listcomp>
    return [self._visit_generic(child) for child in node]
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)

  <.... SKIPPED ....>

  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 54, in _visit
    visited = self._visit_generic(value)
  File "astroid/transforms.py", line 67, in _visit_generic
    return self._visit(node)
  File "astroid/transforms.py", line 57, in _visit
    return self._transform(node)
  File "astroid/transforms.py", line 38, in _transform
    if predicate is None or predicate(node):
  File "astroid/brain/brain_builtin_inference.py", line 141, in _builtin_filter_predicate
    and node.root().name == "re"
  File "astroid/nodes/node_ng.py", line 371, in root
    return self.parent.root()
  File "astroid/nodes/node_ng.py", line 371, in root
    return self.parent.root()
  File "astroid/nodes/node_ng.py", line 371, in root
    return self.parent.root()
  [Previous line repeated 319 more times]
RecursionError: maximum recursion depth exceeded

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "pylint/lint/pylinter.py", line 730, in _get_asts
    ast_per_fileitem[fileitem] = self.get_ast(
  File "pylint/lint/pylinter.py", line 1066, in get_ast
    raise astroid.AstroidBuildingError(
astroid.exceptions.AstroidBuildingError: Building error when trying to create ast representation of module 't'

Expected behavior

It should not crash.

Pylint version

pylint 2.17.4
astroid 2.15.5
Python 3.10.8 (main, Mar  9 2023, 10:11:52) [GCC 12.2.0]

OS / Environment

No response

Additional dependencies

No response
@yilei yilei added the Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling label Jul 11, 2023
@DanielNoord
Copy link
Collaborator

I never know with these kind of issues. pylint probably shouldn't crash but we also shouldn't try to be too sensical about it. I think it is fine to notify the user that their code is unparseable, after all we are a linter.. 😄

Curious to see what others think.

@yilei
Copy link
Contributor Author

yilei commented Jul 11, 2023

Ideally, there should be a way to tell the linter, hey, please ignore this line (pylint: disable=), so it can still check the rest of the file.

At minimum, # pylint: skip-file should work.

@Pierre-Sassoulas Pierre-Sassoulas added Crash 💥 A bug that makes pylint crash Needs decision 🔒 Needs a decision before implemention or rejection and removed Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling labels Aug 16, 2023
@Pierre-Sassoulas
Copy link
Member

I think we decided at some point that explicitely failing is better than silentely failing the inference and having a poorer result without any explanation. At least here it's possible to increase the recursion limit. Skipping the parsing conditionally will be an enormous feature. Right now skip file mean skipping the linting so it would also need a new concept (like '# astroid: no-inference')

@Pierre-Sassoulas
Copy link
Member

Thought about it some more, I think we need to transform all crashes into fatal level messages. There's no reason to prevent the remainder of the analysis to take place when there's a crash, and it's still possible to report the crash when it happens (the message being essentially the same but instead of crashing pylint continue working if possible). That way it's possible to ignore the fatal with the disable when they are expected (until the crash is fixed in pylint). I already had some project were pylint was unusable because of a crash related to pandas, nothing to do but to wait for a fix, it's annoying.

@Pierre-Sassoulas Pierre-Sassoulas mentioned this issue Aug 16, 2023
Closed
@Pierre-Sassoulas Pierre-Sassoulas added this to the 4.0.0 milestone Sep 25, 2023
@jacobtylerwalls jacobtylerwalls mentioned this issue Oct 13, 2023
Closed
2 tasks
@elanor-sparx elanor-sparx mentioned this issue Jan 25, 2024
Closed
@jacobtylerwalls jacobtylerwalls mentioned this issue Feb 18, 2024
Closed
@jacobtylerwalls
Copy link
Member

jacobtylerwalls commented Feb 18, 2024 •
edited
Loading

There are several places where we could try to trap RecursionError and reraise it as a fatal (but ignorable) message.

  1. Inference system
  2. Performing astroid transforms (this issue)

@jacobtylerwalls
Copy link
Member

pylance doesn't crash, but it does admit it can't parse. We can do something similar.

Maximum parse depth exceeded; break expression into smaller sub-expressions: Pylance

@jacobtylerwalls jacobtylerwalls mentioned this issue Feb 18, 2024
Merged
@jacobtylerwalls jacobtylerwalls modified the milestones: 4.0.0, 3.1.0 Feb 18, 2024
@jacobtylerwalls jacobtylerwalls removed the Needs decision 🔒 Needs a decision before implemention or rejection label Feb 18, 2024
@cdce8p cdce8p mentioned this issue Feb 23, 2024
Merged
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Feb 25, 2024
@0-wiz-0
py-astroid: update to 3.1.0.
f25b87d 
    Include PEP 695 (Python 3.12) generic type syntax nodes in get_children(),
    allowing checkers to visit them.
    Refs pylint-dev/pylint#9193
    Add __main__ as a possible inferred value for __name__ to improve
    control flow inference around if __name__ == "__main__": guards.
    Closes #2071
    Following a deprecation period, the names arg to the Import constructor and
    the op arg to the BoolOp constructor are now required, and the doc args
    to the PartialFunction and Property constructors have been removed (call
    postinit(doc_node=...) instead.)
    Following a deprecation announced in astroid 1.5.0, the alias AstroidBuildingException is removed in favor of AstroidBuildingError.
    Include modname in AST warnings. Useful for invalid escape sequence warnings
    with Python 3.12.
    RecursionError is now trapped and logged out as UserWarning during astroid node transformations with instructions about raising the system recursion limit.
    Closes pylint-dev/pylint#8842
    Suppress SyntaxWarning for invalid escape sequences on Python 3.12 when parsing modules.
    Closes pylint-dev/pylint#9322
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Crash 💥 A bug that makes pylint crash
Projects
None yet
Milestone
3.1.0
4 participants
@yilei @Pierre-Sassoulas @DanielNoord @jacobtylerwalls