Implement per-file strict Optional #3206

ddfisher · 2017-04-21T04:32:24Z

I don't think this is quite 100%, but I think it's close. In particular, suggestions for more tests would be very helpful.

This definitely feels a little hacky, but I'm not convinced there's much of a better way to do it. It currently runs cleanly against the smaller of our internal repos, and with only 6 errors against the larger.

gvanrossum

What do you think of the idea of moving the context managers into the call sites? Much less code churn, which somehow is always high on my list of PR quality issues.

gvanrossum · 2017-04-22T23:14:51Z

mypy/checker.py

-            with self.binder.top_frame_context():
-                for d in self.tree.defs:
-                    self.accept(d)
+        with experiments.strict_optional_set(self.options.strict_optional):


I wonder if it would be better to wrap the call sites in such a context manager? Fewer lines in the diff, and there's only one call site in build.py (and another in server/update.py which is only used by fine-grained incrementalism). Ditto for check_second_pass() and visit_file() -- these are each only called from one place in build.py and perhaps another place in server/.

I'm not sure how likely it is that we ever add more call sites, but I think that's a bug in the making. The context manager should always wrap this code -- I don't think there's any case where we'd want to call this while ignoring the per-file Strict Optional setting. I understand that you're not a huge fan of code churn, but I don't think it'd be a worthwhile tradeoff in this instance.

Also, FWIW git blame has the -w flag which ignores whitespace changes, so this doesn't have to mess up anyone's blame.

Hm. git might have that flag but GitHub doesn't (as shown in this code review).

Maybe you can make it a decorator containing a context manager?

It looks like that won't quite work, but apparently github has a -w equivalent too! https://github.com/python/mypy/pull/3206/files?w=1

Hm, the -w flag is flawed -- it doesn't show review comments, and it's not sticky.

gvanrossum · 2017-04-22T23:16:34Z

mypy/experiments.py

+    saved = STRICT_OPTIONAL
+    STRICT_OPTIONAL = value
+    yield
+    STRICT_OPTIONAL = saved


I know you said it was hacky, but this is where I realized quite how hacky... :-)

gvanrossum · 2017-04-22T23:25:41Z

test-data/unit/check-class-namedtuple.test

@@ -407,7 +407,7 @@ class HasNone(NamedTuple):
    x: int
    y: Optional[int] = None

-reveal_type(HasNone(1))  # E: Revealed type is 'Tuple[builtins.int, builtins.int, fallback=__main__.HasNone]'
+reveal_type(HasNone(1))  # E: Revealed type is 'Tuple[builtins.int, Union[builtins.int, builtins.None], fallback=__main__.HasNone]'


Off-topic: I so wish that Union[X, None] were rendered as Optional[X]... (Hm, somewhere below there is a test that shows that: test-data/unit/check-isinstance.test -- what's the difference?)

It is in most cases, actually. Mypy has 2 different ways of rendering types: an internal representation and an external representation. The external representation is what's used for all errors, and renders Union[X, None] as Optional[X]. The internal representation is used by reveal_type for some reason that I no longer recall, and does not do this translation. I've been wanting to clean this up since forever (and make them all have a bit more PEP 484 style), but haven't ever had the time.

ilevkivskyi · 2017-04-23T11:11:52Z

What are the plans concerning --strict-optional in the roadmap? I think it is important to make it default soon. I could volunteer to make necessary changes to annotations in mypy to make mypy --strict-optional mypy pass (if this will help).

gvanrossum · 2017-04-23T17:51:43Z

Regarding the roadmap, we all want this to become the default, but there are several roadblocks.

mypy itself is not easy to make strict-optional-clean (we tried when we first started).
Our (Dropbox) internal codebases are no strict-optional-clean by a long shot.
Ditto for other large codebases we know of.

It would be acceptable to turn on the option by default only when we at least have decent tactics to prevent regressions in mypy itself and the Dropbox codebases. For mypy, that tactic could be to enable it only for a few files, and then gradually increase coverage. That's relatively straightforward (unlike actually getting 100% coverage).

For the Dropbox codebases it's a little more complicated, because we are already actively using the old strict-optional settings in some places, and we have a complicated setup where we run mypy twice, once without strict-optional, and once with it, but suppressing errors in those files that aren't yet strict-optional-clean. I want us to switch to the new approach represented by this PR first before we can even think of making it the default in mypy, just so we don't have a flag that the whole world uses except us (there would be serious quality concerns there).

Really, you could argue with all of this (why should Dropbox be the bar) but I'd like to have some more real-world experience before turning it on by default.

If you find it simple to make mypy itself strict-optional-clean, please be our guest, but hopefully you can submit many small PRs rather than huge one that causes merge conflicts everywhere else.

ilevkivskyi · 2017-04-23T21:05:34Z

@gvanrossum

If you find it simple to make mypy itself strict-optional-clean, please be our guest, but hopefully you can submit many small PRs rather than huge one that causes merge conflicts everywhere else.

This is what I was thinking about. There are in total 547 --strict-optional errors in mypy. PR #3228 fixes 26 errors by modifying 46 lines. After another 20 PRs like #3228 we will have clean --strict-optional.

gvanrossum · 2017-04-23T22:07:41Z

This is what I was thinking about. There are in total 547 --strict-optional errors in mypy. PR #3228 fixes 26 errors by modifying 46 lines. After another 20 PRs like #3228 we will have clean --strict-optional.

Hmm. Sometimes fixing an error in one place makes several new errors appear. Yet I am optimistic.

gvanrossum

Can you also make all the tests pass?

gvanrossum · 2017-04-25T19:52:35Z

mypy/checker.py

@@ -1459,6 +1461,9 @@ def check_multi_assignment(self, lvalues: List[Lvalue],
        rvalue_type = self.expr_checker.accept(rvalue)  # TODO maybe elsewhere; redundant
        undefined_rvalue = False

+        if isinstance(rvalue_type, UnionType) and len(rvalue_type.relevant_items()) == 1:
+            rvalue_type = rvalue_type.relevant_items()[0]


I don't like how this is calling relevant_items() twice.

Yeah, that was something that I was thinking about as I was writing this. I could change this to:

if isinstance(rvalue_type, UnionType): relevant_items = rvalue_type.relevant_items() if len(relevant_items) == 1: rvalue_type = relevant_items[0]

but I'm not sure that's particularly nicer? If it seems better to you, then I'm happy to do it.

Yeah, I like that better (the if condition was getting a bit long anyways).

gvanrossum · 2017-04-25T19:52:52Z

mypy/checker.py

-                rvalue_type = cast(TupleType, self.expr_checker.accept(rvalue, lvalue_type))
+                reinferred_rvalue_type = self.expr_checker.accept(rvalue, lvalue_type)
+                if (isinstance(reinferred_rvalue_type, UnionType) and
+                        len(reinferred_rvalue_type.relevant_items()) == 1):


Again, this calls relevant_items() twice.

gvanrossum · 2017-04-25T19:53:44Z

mypy/checker.py

@@ -1486,7 +1491,12 @@ def check_multi_assignment_from_tuple(self, lvalues: List[Lvalue], rvalue: Expre
            if not undefined_rvalue:
                # Infer rvalue again, now in the correct type context.
                lvalue_type = self.lvalue_type_for_inference(lvalues, rvalue_type)
-                rvalue_type = cast(TupleType, self.expr_checker.accept(rvalue, lvalue_type))
+                reinferred_rvalue_type = self.expr_checker.accept(rvalue, lvalue_type)
+                if (isinstance(reinferred_rvalue_type, UnionType) and


Can you add a comment here about what's going on? I think this may be the crucial place where Optional[X] becomes X?

gvanrossum · 2017-04-25T19:59:08Z

mypy/types.py

+                   for x in self.relevant_items())
+
+    def relevant_items(self) -> List[Type]:
+        """Returns all Union items relevant to current context.  In particular,


I think it would be better if you just wrote that this function removes None from unions when strict None checking is off. (And "context" is a technical term in most of mypy so that word threw me off for the first few milliseconds.)

Yeah, that's definitely clearer.

ddfisher · 2017-04-25T22:42:16Z

🎉!

I think this still needs a little bit more work before merging -- a few more places need tweaking so it doesn't cause errors in our internal codebase. I'll ping here again once I've done that!

gvanrossum

I really would like to make another plea for doing the proper refactor (enabling the decorator-based solution) now rather than piling up tech debt.

Highly indented code is less readable.

You could make two decorators, one that uses self.options, the other uses options passed in.

JukkaL · 2017-05-02T16:39:26Z

test-data/unit/check-flags.test

+x = 0  # type: Optional[int]
+y = None  # type: None
+
+[file mypy.ini]


Would it make sense to add tests with invariant collections such as List? In non-strict-optional files, should List[Optional[int]] be compatible with List[int] and vice versa?

ilevkivskyi · 2017-05-04T22:49:25Z

Just an interesting observation: 6 --strict-optional errors were introduced by recent PRs in files that I previously cleaned (there are 20 such files now). None creeps in much faster than I expected :-)

gvanrossum · 2017-05-26T15:22:30Z

@ilevkivskyi wrote

errors were introduced by recent PRs in files that I previously cleaned

You can guard against that by adding per-module strict_optional = True settings to mypy_strict_optional.ini.

@ddfisher Can you resolve the conflicts and address the reviews?

ilevkivskyi · 2017-06-20T22:15:13Z

@gvanrossum

You can guard against that by adding per-module strict_optional = True settings to mypy_strict_optional.ini.

Do I understand correctly that will only work after this PR is merged? Or can I continue working on making mypy strict-optional-clean without waiting for this PR?

gvanrossum · 2017-06-20T23:13:05Z

Sorry, yes, that option will only work once this PR is merged. We apologize for the delay, it's been on the back burner since around PyCon, but the plan is to finish the work before July starts.

ddfisher · 2017-06-24T00:02:38Z

(Not yet ready to merge -- there are several errors in internal codebases that still need to be fixed.)

gvanrossum

I am fine with this landing.

gvanrossum · 2017-06-27T19:16:03Z

test-data/unit/check-flags.test

+
+[file optional.py]
+from typing import Optional, List
+def f(x: List[int]) -> None: pass


Didn't you mean to involve f() in some test in this file?

I think I left it for parallel structure with the other tests, but it doesn't actually participate in this one.

ddfisher · 2017-06-28T22:34:14Z

This should now be ready to land. It still causes a small number of errors across our two main internal codebases, but the errors are all legitimate (and it's not clear to me why some of them aren't caught currently).

gvanrossum · 2017-06-28T23:26:50Z

If there's an issue requesting this feature can you track it down and close it?

ddfisher · 2017-06-28T23:27:21Z

🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉

ddfisher · 2017-06-28T23:30:52Z

I don't see an issue for this, actually.

Implement per-file strict Optional

454844a

gvanrossum reviewed Apr 23, 2017

View reviewed changes

gvanrossum requested changes Apr 25, 2017

View reviewed changes

ddfisher added 2 commits April 25, 2017 15:22

fix tests; tweak docstring

61cf01b

rewrite ifs; add comments

e094b8d

gvanrossum approved these changes Apr 25, 2017

View reviewed changes

gvanrossum requested changes Apr 26, 2017

View reviewed changes

This was referenced Apr 26, 2017

[clean strict optional] Fix some strict optional errors in mypy #3228

Merged

Release mypy 0.510 #3273

Closed

JukkaL reviewed May 2, 2017

View reviewed changes

ddfisher added 2 commits June 23, 2017 16:43

Merge branch 'master' into per-file-strict-optional

8d1623d

Add List tests

0ed98c6

fix lint

1b5fb11

gvanrossum approved these changes Jun 27, 2017

View reviewed changes

ddfisher added 3 commits June 28, 2017 15:26

Fix accidental typeshed update

1df4c7d

Add missing relevant_items

d2a4eb6

Fix last known bug

8e40de0

gvanrossum merged commit 0619e26 into master Jun 28, 2017

gvanrossum deleted the per-file-strict-optional branch June 28, 2017 23:26

ddfisher mentioned this pull request Jun 28, 2017

Improvements to --strict-optional[-whitelist] #2126

Closed

Implement per-file strict Optional #3206

Implement per-file strict Optional #3206

Conversation

ddfisher commented Apr 21, 2017 • edited Loading

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented Apr 23, 2017

gvanrossum commented Apr 23, 2017

ilevkivskyi commented Apr 23, 2017

gvanrossum commented Apr 23, 2017

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddfisher commented Apr 25, 2017

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented May 4, 2017

gvanrossum commented May 26, 2017

ilevkivskyi commented Jun 20, 2017

gvanrossum commented Jun 20, 2017

ddfisher commented Jun 24, 2017

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddfisher commented Jun 28, 2017

gvanrossum commented Jun 28, 2017

ddfisher commented Jun 28, 2017

ddfisher commented Jun 28, 2017

ddfisher commented Apr 21, 2017 •

edited

Loading