Warn about @contextlib.contextmanager without try/finally in generator functions

[ghost]

Is your feature request related to a problem?

If you decorate a generator function with @contextlib.contextmanager and use the context manager cm inside a generator function, the context manager's __exit__ will not be always be called because the generator raises GeneratorExit, which ends the generator that defines cm.

Notice that cm exit never appears, as cm's __exit__ is never called.

from __future__ import generator_stop

import contextlib
import weakref
import sys


@contextlib.contextmanager
def cm():
    print("cm enter")
    a = yield
    print('cm exit')

def genfunc():
    with cm():
        print("stepping")
        yield

def main():
    gen = genfunc()
    ref = weakref.ref(gen, print)
    next(gen)


if __name__ == "__main__":
    main()

Output:

cm enter
stepping
<weakref at 0x7f4739e38e08; dead>

Describe the solution you'd like

Generator functions decorated with @contextlib.contextmanager should not be used in generators unless the context manager has finally:

@contextlib.contextmanager
def cm():
    try:
        yield
    finally:
        # cleanup code here
        ...

or specifically handles GeneratorExit:

@contextlib.contextmanager
def cm():
    try:
        yield
    except GeneratorExit:
        # cleanup code here
        ...

Additional context

Add any other context about the feature request here.

[PCManticore]

Thank you @apnewberry This makes sense.

[ghost]

Perhaps optionally a stricter version of this check: any @contextlib.contextmanager should contain

@contextlib.contextmanager
def f():
    ...
    try:
        yield
    finally:
        ...

regardless of whether it's used in a generator, since the client code that uses the context manager in a generator may not be using pylint, and the context manager should provide the right behavior for such callers.

[rhyn0]

If there is desire to get this moving, I am available and just have some clarifying questions.

Same code as supplied by issue opener yields the following in Python 3.11.

$ python3 --version
  Python 3.11.5
$ python3 test.py
   cm enter
   stepping
   <weakref at 0x103076f70; dead>

Any guesses on the error name to use? missing-exception-handler-contextmanager ?

And I'm guessing this should be added into 'extensions' for a bad builtin?

Finally, it seems like we should only raise the error if a contextlib.contextmanager decorated function has no error handling for GeneratorExit - agreed?

[Pierre-Sassoulas]

Thank you for specifying the design, clearly this one was not PR ready. And thank you for contributing to pylint :)

Regarding the naming, we have a message named yield-inside-async-function and yield-outside-function, it I considered yield-inside-context-manager but I don't think it fits very well (not all yield are bad in context manager).

Admittedly the message is hard to name succinctly (there's generator, context manager and unhandled exception, 5 concepts when we generally have 4 words messages names or less) but maybe one of the following ?

• contextmanager-yield-unhandled-exception
• contextmanager-generator-missing-cleanup / contextmanager-generator-missing-exit
• contextmanager-yield-before-exit / contextmanager-yield-before-cleanup

This can be changed with a search&replace before merging if someone has a great name.

About putting it in an extension, I'm not sure. Code that is implicitly not run is at least warning level, and warnings are generally valuable enough to be put in the basic checker. (Similar to return-in-finally for example). This can also changed easily, please create a new file like https://github.com/pylint-dev/pylint/blob/main/pylint/checkers/base/pass_checker.py so it can be moved easily if we simply change _BasicChecker to a BaseChecker if we want to move to an extension.

Finally, it seems like we should only raise the error if a contextlib.contextmanager decorated function has no error handling for GeneratorExit - agreed?

And I think it should also contains a yield (is a generator)

[rhyn0]

I like the sound of contextmanager-generator-missing-cleanup seems to hit on the 2 things that define this behavior and the issue with the code.

This can also changed easily, please create a new file like https://github.com/pylint-dev/pylint/blob/main/pylint/checkers/base/pass_checker.py so it can be moved easily if we simply change _BasicChecker to a BaseChecker if we want to move to an extension.

This sounds good

Finally, it seems like we should only raise the error if a contextlib.contextmanager decorated function has no error handling for GeneratorExit - agreed?

And I think it should also contains a yield (is a generator)

Yep that is my bad for typo - will see what I can get going on a first pass.

[rhyn0]

@Pierre-Sassoulas I think this works for the given cases, but there are examples where it raises the error when I think it should probably not. Example:

import contextlib

@contextlib.contextmanager
def test():  # [contextmanager-generator-missing-cleanup]
  yield 2

In simple functions where there is no cleanup, this checker will still raise the Error. But want to check in and make sure the direction is right.

[DanielNoord]

I haven't seen this issue before, but I agree. This code is fine and I don't really see what this message is trying to achieve.

The original code is just wrong:

from __future__ import generator_stop

import contextlib
import sys
import weakref


@contextlib.contextmanager
def cm():
    print("cm enter")
    a = yield
    print("cm exit")


def genfunc():
    with cm():
        print("stepping")
        yield


def main():
    gen = genfunc()
    ref = weakref.ref(gen, print)
    list(iter(gen))


if __name__ == "__main__":
    main()

Changing the next to iter does execute the part after yield. It's expected that if you only call next once that you stop execution at the yield statement. I don't see what this message should be warning about and I think the PR already shows that there are a lot of "false positives" in the primer.

[rhyn0]

Do you think there should be a warning for this at all then? Unclear why OP necessarily needed their code to look like that, but is it good to still include this error as a catch for people who might not know of this side effect?

I could see myself writing something similar to this using a DBAPI connection and cursor if I tried. I think there is value in warning for original case

[DanielNoord]

I don't think it is feasible to warn for the original code without creating too many false positives.
What bad is about the original code? The fact that the iterator wasn't exhausted? That seems like a way too broad check.

[rhyn0]

My thinking would be that a context manager decorated generator function that has cleanup code may or may not have cleanup properly executed.

The problem for me is that the distinguishing is as you said, whether the user of the function knows which iterator consumables to use.

[jacobtylerwalls]

Apparently, even if the suggestion of the proposed checker is implemented, you still will have "cleanup code" executed at the wrong time if you use the context manager as a decorator instead of as a with statement. See cpython documentation issue.

This suggests to me that "cleanup code" in a context manager that needs to consider this try/finally pattern can only be distinguished from "cleanup code" that may never "clean up" if used in a partially-exhausted generator (okay) if the decorator is managing a resource (e.g. yielding something other than None, or possibly a truthy constant).

If we want to continue with this check, maybe we can limit it to just:

• warning on the generator functions, not on the context manager itself
• only if a with/async with statement is used
• only if the val in with cm() as val is not discarded, e.g. not for uses like with cm():
• only if the context manager yields something other than None, or possibly other than a truthy constant
• only if the context manager lacks a finally, or does not catch GeneratorExit

That will hopefully reduce the messages detected in the draft PR primer run.

@rhyn0 what do you think?

[jacobtylerwalls]

A python discussion about how this (yielding out of a context manager managing a resource) is a bad pattern, and could possibly lead to a PEP at some point, but in the meantime, could a static checker help out?

So I think doing something here would be nice. cc/ @DanielNoord

[rhyn0]

After reading the linked discussion about Trio's use case, I see a better direction for this warning. Having warnings on the generator, when it yields non constant values, is probably best path forward.

If we want to incorporate checks for code usage of these generators at with/async with I think that would be the points where a warning/reminder would be the most helpful. So I agree with the more specific checks from @jacobtylerwalls

[DanielNoord]

Okay, let's move forward with the more specific checks!

[rhyn0]

@jacobtylerwalls @DanielNoord pushed some changes but wanted to come back to this to make sure the Good/Bad examples make sense as to what we laid out.

# good.py
import contextlib


@contextlib.contextmanager
def good_cm_except():
    contextvar = "acquired context"
    print("good cm enter")
    try:
        yield contextvar
    except GeneratorExit:
        print("good cm exit")


def genfunc_with_cm():
    with good_cm_except() as context:
        yield context * 2


def genfunc_with_discard():
    with good_cm_except():
        yield "discarded"


@contextlib.contextmanager
def good_cm_yield_none():
    print("good cm enter")
    yield
    print("good cm exit")


def genfunc_with_none_yield():
    with good_cm_yield_none() as var:
        print(var)
        yield "discarded"


@contextlib.contextmanager
def good_cm_finally():
    contextvar = "acquired context"
    print("good cm enter")
    try:
        yield contextvar
    finally:
        print("good cm exit")


def good_cm_finally_genfunc():
    with good_cm_finally() as context:
        yield context * 2

# bad.py
import contextlib


@contextlib.contextmanager
def cm():
    contextvar = "acquired context"
    print("cm enter")
    yield contextvar
    print("cm exit")


def genfunc_with_cm():  # [contextmanager-generator-missing-cleanup]
    with cm() as context:
        yield context * 2

I think this hits all the bullet points we said earlier, but if there are any misses on this please let me know

[jacobtylerwalls]

Overall looks good.

I'd add some good cases with decorators e.g. @cm() that would otherwise be bad cases if they were redone as with cm(): statements. (good is a misnomer; still a bad pattern probably, but not worth warning about as they're unfixable with the prescription we're offering, which is to add cleanup code)

I think the one question I have is about the difference between cm() in bad.py versus good_cm_yield_none() in good.py. It seems the difference is whether None is yielded versus a constant. I'm okay with that-- in order to keep things simple-'n-stupid--but I have done things like yield True when I probably should have just done yield None. A boolean literal like True or False doesn't need to be cleaned up. We could still keep things pretty simple-'n-stupid by just allowing literal None, True, and False as the things you can yield without triggering this message. Thoughts? I'm not of strong feelings on it.

[jacobtylerwalls]

Ah, I see in the PR are you allowing yielding constants (as opposed to names) in addition to None as legal yields. So looking good, I'd just add a test case to good.py for it.

[eli-schwartz]

If I have code that is decorated with @contextmanager, creates with TemporaryDirectory() and does some prep work, then exits the function by yielding a file in the temporary directory -- there is no expected code to run after yield.

This check seems to indicate that I should explicitly wrap it in try/finally with cleanup code consisting of "pass"?

[jacobtylerwalls]

In #9625 we're discussing the case you bring up, where there is no code after yield.