Ignoring a file with a single # type: ignore comment. #6830

brandtbucher · 2019-05-14T04:03:35Z

Fixes #626. For background on the design choices, here is what our current ASTs allow us to find:

In all versions of Python, we can accurately detect bare # type: ignore comments before the first statement in a module.
In Python 3.8+, we can accurately detect the placement of bare # type: ignore comments within a list of statements (we need end_lineno to do this properly).
In no current version of Python can we accurately detect bare # type: ignore comments before the first statement in an indented code block using the AST alone.

Considering these limitations, the behavior I chose here is:

All versions of Python will ignore a whole file if it contains a bare # type: ignore comment before its first statement.
Python 3.8+ will ignore all statements following a bare # type: ignore comment at the top level of a module. This is now a separate PR: Ignoring the rest of a file with a single # type: ignore comment on Python 3.8+. #6841.
Bare # type: ignore comments have no special meaning inside of indented code blocks. If we were to allow this, we would need to do one of the following (and I personally don't like any of them):
- Not recognize them at the start of a block. 🙁
- Guess where the for/while/with/class/def/try bit ends, a la decorated definitions prior to Python 3.8. 🙁
- Parse the raw source of the file ourselves to see where the above block-starters end. 🙁
- Modify ast parsing to tell us where the block-starters end. 🙁

A few notes:

Ignoring is accomplished by nesting all remaining statements in an unreachable Block.
ASTConverter.extra_type_ignores: List[int] is now ASTConverter.type_ignores: Set[int] and is populated before the AST is converted.
Tests are added in check-ignore.test ~~and check-38.test~~.

brandtbucher · 2019-05-14T04:05:17Z

This only partially satisfies PEP 484's spec, but I still think it's a good start:

A # type: ignore comment on a line by itself is equivalent to adding an inline # type: ignore to each line until the end of the current indented block. At top indentation level this has effect of disabling type checking until the end of file.

In my opinion, the above wording is a bit too general. For one, it technically allows placing these ignores in weird places, like blank lines in the middle of expressions, or between a decorator and a function/class def. Taken literally, it also makes --warn-unused-ignores behavior super annoying, since every single line in the block now has a virtual # type: ignore comment.

gvanrossum

Great work! I have some suggestions but I think this is on the short road to success.

Could you add docs as well? Perhaps add something about ignoring a whole file somewhere in common_issues.rst.

gvanrossum · 2019-05-14T17:09:08Z

mypy/fastparse.py

+        return n.lineno
+
+    def translate_stmt_list(self,
+                            l: Sequence[ast3.stmt],


Please choose a longer name for this argument, maybe stmts? (I know the original code uses one-letter names, but I don't want to encourage the practice for new code, and you've added a lot here.)

gvanrossum · 2019-05-14T17:10:24Z

mypy/fastparse.py

+
+        if module and l and self.type_ignores and min(self.type_ignores) < self.get_line(l[0]):
+            self.errors.used_ignored_lines[self.errors.file].add(min(self.type_ignores))
+            b = Block(self.fix_function_overloads(self.translate_stmt_list(l)))


Maybe rename to block?

gvanrossum · 2019-05-14T17:12:25Z

mypy/fastparse.py

+
+    def translate_stmt_list(self,
+                            l: Sequence[ast3.stmt],
+                            module: bool = False) -> List[Statement]:


Perhaps rename to ismodule?

gvanrossum · 2019-05-14T17:12:43Z

mypy/fastparse.py

+
+        for i, e in enumerate(l):
+
+            if module:  # and...


Somehow it looks inefficient to check for module and sys.version_info each time through the loop. Also the comment here adds nothing. You did know and short-circuits, right? You can combine the two checks and save one level of indentation.

It was originally if module and sys.version_info >= (3, 8):, but the mypy self check complained when I accessed e.end_lineno on 336. As I understand it, the mypy's version checks don't work for compound tests like this? Splitting it up silences the error, and I believe the emitted bytecode is identical.

If it would be clearer, I can just # type: ignore line 336 instead!

OK, then the two if statements is optimal -- just put in a clearer comment explaining that this is to work around a mypy issue.

gvanrossum · 2019-05-14T17:17:33Z

mypy/fastparse.py

-        for e in l:
+        line = 0
+
+        for i, e in enumerate(l):


Again, I'd propose a longer name for e, e.g. stmt. (But, i for the index is traditional, so please keep that.)

gvanrossum · 2019-05-14T17:18:34Z

mypy/fastparse.py

+                if sys.version_info >= (3, 8):
+
+                    # In Python 3.8, a "# type: ignore" comment between statements at
+                    # the top level of a module skips checking for everything else:


Most importantly, explain why: this check needs access to end_lineno which only exists in 3.8 and up.

gvanrossum · 2019-05-14T17:19:31Z

mypy/fastparse.py

+    def translate_stmt_list(self,
+                            l: Sequence[ast3.stmt],
+                            module: bool = False) -> List[Statement]:
+


I'd leave this blank line out, as well a the one below this block comment. (Ditto for other block comments below; in general you sprinkle more whitespace throughout the code than is mypy's tradition.)

I've removed a lot of my extra whitespace throughout.

gvanrossum · 2019-05-14T17:30:29Z

mypy/fastparse.py

@@ -294,11 +294,50 @@ def translate_expr_list(self, l: Sequence[AST]) -> List[Expression]:
            res.append(exp)
        return res

-    def translate_stmt_list(self, l: Sequence[AST]) -> List[Statement]:
+    def get_line(self, n: ast3.stmt) -> int:


Maybe rename to get_lineno? Without context, the name get_line vaguely makes me think that it would get the source code line somehow.

gvanrossum · 2019-05-14T17:32:42Z

mypy/fastparse.py

        return MypyFile(body,
                        self.imports,
                        False,
-                        set(ignores),
+                        set(self.type_ignores),


I'd add a comment explaining why it's copied here. (If there is a good reason!)

Looks like this isn't needed. Removed.

brandtbucher · 2019-05-14T20:37:32Z

@gvanrossum I've added docs and addressed your concerns. Tests are passing, ready for another review!

gvanrossum

Almost there! I just wish the doc section to be moved a bit.

gvanrossum · 2019-05-15T18:18:04Z

docs/source/common_issues.rst

@@ -131,6 +131,30 @@ The second line is now fine, since the ignore comment causes the name
    if we did have a stub available for ``frobnicate`` then mypy would
    ignore the ``# type: ignore`` comment and typecheck the stub as usual.

+A ``# type: ignore`` comment at the top of a module (before any statements,


This hardly qualifies as "locally" silencing the checker, so I think it doesn't belong in this section.

I would make it a new section titled "Ignoring a whole file" to be inserted after this section (i.e. just before the section named "Unexpected errors about 'None' and/or 'Optional' types").

I've broken it out, as requested.

JelleZijlstra · 2019-05-15T21:20:01Z

docs/source/common_issues.rst

+    import frobnicate
+    frobnicate.start()
+
+When running mypy with Python 3.8 or later, a ``# type: ignore`` comment


This doesn't seem like a very useful behavior to me; when would you actually want this?

It's also potentially error-prone. In the example below, a user could write the # type: ignore expecting to silence errors on the import frobnicate line, but then get confused when this actually silences all errors in the whole file.

For context, pyre's behavior is for an ignore comment on a line by itself to suppress errors on only the next line: https://pyre-check.org/docs/error-suppression.html#explicitly-suppressing-errors.

expecting to silence errors on the import frobnicate line

Why would you expect that to work?

Also, the behavior implemented here is specified by PEP 484.

I had assumed that there was some degree of consensus between discussion on #626 and the relevant section of PEP 484 that a # type: ignore on its own line should disable type checking "until the end of the current indented block". This PR is an attempt to bring mypy closer to that spec. I don't have a personal use case for it, but I assume the PEP's authors gave it some thought? Maybe it's worth adding a note or some other warning to the user to help avoid mistakes.

One scenario I can imagine is somebody working their way through an existing code base with several large modules, adding or correcting annotations as they go. Test code also comes to mind, where you may have several fixtures you want type-checked near the top of the module, but intentionally broken code after that.

If we decide that the ignore-rest-of-file-on-3.8 part of this PR should be removed or deferred, I'm fine with that, though. I added it because I could for 3.8, but I'm certainly not married to it.

Why would you expect that to work?

It's the intuitively more obvious behavior to me (which, granted, is not a very strong argument), and it's also what Pyre implemented. On the other hand, pytype's behavior is closer to what PEP 484 specifies, but with a way to turn type checking back on: https://github.com/google/pytype/blob/master/docs/user_guide.md#silencing-errors.

Also, the behavior implemented here is specified by PEP 484.

That's true, but this is a part of PEP 484 that has remained unimplemented in mypy for five years. As far as I can tell there have been few if any user requests for the full behavior specified in PEP 484—users want to be able to ignore the whole file, but I don't see people asking for "ignore errors from an arbitrary point to the end of the file". So I think it's worth asking if the PEP 484 behavior really is what we want. We can always change PEP 484 too.

If #2938 is implemented, even using # type: ignore to ignore the whole file could be a redundant feature.

Also, I'm happy to take this to the python/typing tracker to discuss potentially changing the PEP. My concern on this PR is that we don't introduce this behavior into mypy too soon and produce backward compatibility problems if we ever decide to change it.

gvanrossum · 2019-05-16T04:34:28Z

I'm surprised this is so controversial, but given that it is, I am okay with taking it to a larger forum (though I have a slight preference for typing-sig, which probably reaches more people).

Note that there is a specific use case that depends on this which we use heavily at Dropbox: a # type: ignore at the very end of a file has no effect on the type checker. (At Dropbox, it results in our internal wrapper script adding the file to the mypy "build" even if it has no other type comments.)

I agree that there's probably no strong use case for putting a # type: ignore in the "middle" of a file.

I don't think that #2938 covers this, since that is mypy-specific; the "lone type-ignore" syntax is meant to be honored by all type checkers (and I feel quite apologetic that it has taken mypy so long to implement it).

brandtbucher · 2019-05-16T05:24:12Z

Thanks for the honest feedback @JelleZijlstra. I agree that something that seems as error prone as this deserves to be thoughtfully considered.

@gvanrossum Should I break out the partial-file behavior into another PR, and leave the whole-file behavior here? It seems like that specific feature is useful and uncontroversial by itself (it’s also been requested enough to be marked as “high priority” for several years).

gvanrossum · 2019-05-16T05:31:03Z

Yes, if you could move all the Python 3.8 stuff out I think it will be less controversial (I hope Jelle will give that his blessing without going to the larger forum).

…

On Wed, May 15, 2019 at 10:24 PM Brandt Bucher ***@***.***> wrote: Thanks for the honest feedback @JelleZijlstra <https://github.com/JelleZijlstra>. I agree that something that seems as error prone as this deserves to be thoughtfully considered. @gvanrossum <https://github.com/gvanrossum> Should I break out the partial-file behavior into another PR, and leave the whole-file behavior here? It seems like that specific feature is useful and uncontroversial by itself (it’s also been requested enough to be marked as “high priority” for several years). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6830?email_source=notifications&email_token=AAWCWMSWABY6XOAAOZMJUXDPVTVYDA5CNFSM4HMVJCZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVQWERQ#issuecomment-492921414>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWCWMTB2NRVWUMXP5WG2GDPVTVYDANCNFSM4HMVJCZQ> .

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

JelleZijlstra · 2019-05-16T05:31:39Z

Yes, I don't have any problem with the behavior where # type: ignore at the top of the file ignores the whole file.

brandtbucher · 2019-05-16T07:43:13Z

@gvanrossum This branch is clean now!

gvanrossum · 2019-05-16T14:27:19Z

OK! Jelle, please start a conversation on whether the language about this in PEP 484 should be changed.

Fixes python#626 (There are potential further features as discussed in the PR, but these are somewhat controversial and may even be excised from PEP 484.)

Fixes #7785. This was a mistake in my original PR that introduced the feature (#6830). I forgot to copy one of the changes in `fastparse.py` to `fastparse2.py`. It's an embarrassingly simple fix. I also included a regression test for good measure.

brandtbucher added 3 commits May 13, 2019 19:48

Stop checking at bare top-level "# type: ignore" comments.

7a51cf2

Test new "# type: ignore" behavior.

aa11089

Appease mypy.

ce54ab4

gvanrossum reviewed May 14, 2019

View reviewed changes

brandtbucher added 2 commits May 14, 2019 13:19

Add docs for new # type: ignore usage.

4567661

Code cleanup from GvR's first review.

c95b98d

gvanrossum reviewed May 15, 2019

View reviewed changes

JelleZijlstra reviewed May 15, 2019

View reviewed changes

Break out a new section for this feature.

07cc315

Back out Python 3.8 feature.

9937b1b

brandtbucher changed the title ~~Ignoring all or part of a file with a single # type: ignore comment.~~ Ignoring a file with a single # type: ignore comment. May 16, 2019

brandtbucher mentioned this pull request May 16, 2019

Ignoring the rest of a file with a single # type: ignore comment on Python 3.8+. #6841

Closed

gvanrossum merged commit c73c95c into python:master May 16, 2019

brandtbucher deleted the ignore-file branch May 16, 2019 18:56

gwk mentioned this pull request May 16, 2019

Whole file # type: ignore does not silence import errors #6852

Closed

brandtbucher mentioned this pull request May 17, 2019

Ignoring the rest of a file with a single # type: ignore comment on Python 3.8+. #6854

Closed

brandtbucher mentioned this pull request Jun 7, 2019

Ignore typing errors in individual blocks #6948

Open

lazka mentioned this pull request Aug 2, 2019

[WIP] Stub generation pygobject/pgi-docgen#176

Open

brandtbucher mentioned this pull request Aug 22, 2019

Python Triage Team Application: Brandt Bucher python/core-workflow#356

Closed

brandtbucher mentioned this pull request Oct 24, 2019

Enable whole-file # type: ignore comments in Python 2.7. #7789

Merged

koenlek mentioned this pull request Apr 9, 2021

Allow error-code specific whole-file # type: ignores #10304

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ignoring a file with a single # type: ignore comment. #6830

Ignoring a file with a single # type: ignore comment. #6830

brandtbucher commented May 14, 2019 •

edited

Loading

brandtbucher commented May 14, 2019

gvanrossum left a comment

gvanrossum May 14, 2019

gvanrossum May 14, 2019

gvanrossum May 14, 2019

gvanrossum May 14, 2019

brandtbucher May 14, 2019

gvanrossum May 14, 2019

gvanrossum May 14, 2019

gvanrossum May 14, 2019

gvanrossum May 14, 2019

brandtbucher May 14, 2019

gvanrossum May 14, 2019

gvanrossum May 14, 2019

brandtbucher May 14, 2019

brandtbucher commented May 14, 2019

gvanrossum left a comment

gvanrossum May 15, 2019

brandtbucher May 15, 2019

JelleZijlstra May 15, 2019

JelleZijlstra May 15, 2019

gvanrossum May 15, 2019

brandtbucher May 15, 2019

JelleZijlstra May 15, 2019 •

edited

Loading

JelleZijlstra May 15, 2019

gvanrossum commented May 16, 2019

brandtbucher commented May 16, 2019

gvanrossum commented May 16, 2019 via email

JelleZijlstra commented May 16, 2019

brandtbucher commented May 16, 2019

gvanrossum commented May 16, 2019

Ignoring a file with a single # type: ignore comment. #6830

Ignoring a file with a single # type: ignore comment. #6830

Conversation

brandtbucher commented May 14, 2019 • edited Loading

brandtbucher commented May 14, 2019

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brandtbucher commented May 14, 2019

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelleZijlstra May 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum commented May 16, 2019

brandtbucher commented May 16, 2019

gvanrossum commented May 16, 2019 via email

JelleZijlstra commented May 16, 2019

brandtbucher commented May 16, 2019

gvanrossum commented May 16, 2019

brandtbucher commented May 14, 2019 •

edited

Loading

JelleZijlstra May 15, 2019 •

edited

Loading