F0002 When trying to infer too many items

[tushar-deepsource]

Bug description

Pylint throws an F0002 while trying to lint a file with large enough inferences, here's a minimal example:

x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30 = foo()

def bar():
    return [
        x0 + x1 + x10 + x100 + x101 + x103 + x104 + x106 + x109 + x11,
        x0 + x1 + x10 + x100 + x101 + x102 + x103 + x104 + x105 + x106 + x107 + x108 + x109 + x11 + x110 + x111 + x112 + x113 + x114 + x115 + x116 + x117 + x118 + x119 + x12 + x120 + x121 + x122 + x123 + x124 + x125 + x126 + x127 + x128 + x129 + x13 + x130 + x131 + x132 + x133 + x134 + x135 + x136 + x137 + x138 + x139 + x14 + x140 + x141 + x142 + x143 + x144 + x145 + x146 + x147 + x148 + x149 + x15 + x150 + x151 + x152 + x153 + x154 + x155 + x156 + x157 + x158 + x159 + x16 + x160 + x161 + x162 + x163 + x164 + x17 + x18 + x19 + x2 + x20 + x21 + x22 + x23 + x24 + x25 + x26 + x27 + x28 + x29 + x3 + x30 + x31 + x32 + x33 + x34 + x35 + x36 + x37 + x38 + x39 + x4 + x40 + x41 + x42 + x43 + x44 + x45 + x46 + x47 + x48 + x49 + x5 + x50 + x51 + x52 + x53 + x54 + x55 + x56 + x57 + x58 + x59 + x6 + x60 + x61 + x62 + x63 + x64 + x65 + x66 + x67 + x68 + x69 + x7 + x70 + x71 + x72 + x73 + x8 + x9
        ]

es = bar()
print([e.foobar() for e in es])

This can be handled and caught in astroid during inference, rather than reporting it to the user.

Configuration

No response

Command used

pylint asd.py

Pylint output

Exception on node <Call l.11 at 0x102a5b4c0> in file '/Users/tusharsadhwani/code/marvin-python/asd.py'
Traceback (most recent call last):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 140, in raise_if_nothing_inferred
    yield next(generator)
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 104, in wrapped
    if context.push(node):
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1395, in infer_all
    return list(node.infer(context=context))
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 169, in infer
    yield from self._infer(context=context, **kwargs)
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 154, in raise_if_nothing_inferred
    yield from generator
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 109, in wrapped
    for res in _func(node, context, **kwargs):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 344, in infer_attribute
    for owner in self.expr.infer(context):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 154, in raise_if_nothing_inferred
    yield from generator
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 109, in wrapped
    for res in _func(node, context, **kwargs):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/bases.py", line 165, in _infer_stmts
    for inf in stmt.infer(context=context):  # type: ignore[union-attr]
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
[..... TRUNCATED .....]
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 874, in _infer_binop
    for lhs, rhs in itertools.product(lhs_iter, rhs_iter):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 127, in yes_if_nothing_inferred
    yield next(generator)
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 109, in wrapped
    for res in _func(node, context, **kwargs):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 532, in _filter_operation_errors
    for result in infer_callable(self, context):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/inference.py", line 874, in _infer_binop
    for lhs, rhs in itertools.product(lhs_iter, rhs_iter):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 182, in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/decorators.py", line 150, in raise_if_nothing_inferred
    raise InferenceError(
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/astroid/exceptions.py", line 277, in __init__
    super().__init__(message, **kws)
RecursionError: maximum recursion depth exceeded while calling a Python object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 90, in walk
    callback(astroid)
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/checkers/base/basic_error_checker.py", line 432, in visit_call
    for inferred in infer_all(node.func):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1399, in infer_all
    raise AstroidError from e
astroid.exceptions.AstroidError
asd.py:1:0: F0002: asd.py: Fatal error while checking 'asd.py'. Please open an issue in our bug tracker so we address this. There is a pre-filled template that you can use in '/Users/tusharsadhwani/Library/Caches/pylint/pylint-crash-2022-12-05-16-15-46.txt'. (astroid-error)

Expected behavior

No F0002

Pylint version

pylint 2.15.7
astroid 2.12.13
Python 3.10.6 (main, Aug 30 2022, 04:58:14) [Clang 13.1.6 (clang-1316.0.21.2.5)]

OS / Environment

No response

Additional dependencies

No response

[Pierre-Sassoulas]

We had this problem several time, but I'm not sure that silently failing all the time with an inference error is better for genuinely large recursions or genuinely big calculation. Maybe we should catch the RecursionError or use a timeout, but raise a warning...

[tushar-deepsource]

Yeah, sounds good to me.

[DanielNoord]

I have often voted against catching RecursionErrors too broadly. Often it is just a sign of poorly written code, although I must confess in this case I don't see an immediate fix.

What doesn't help is that I think the calls to all the decorators that we use in astroid make us hit the error sooner?

[jacobtylerwalls]

This specific example no longer reproduces on pylint 3.0.3. We have other open issues about recursion where we can take up the question of what to do, e.g. #8842.

I have often voted against catching RecursionErrors too broadly. Often it is just a sign of poorly written code, although I must confess in this case I don't see an immediate fix.

We want pylint to be used on code that needs help. It won't ever be adopted if it crashes when people try to first use it. The fact that astroid uses recursive algorithms is an implementation detail we shouldn't blame on the code being linted.