More precise return types for open(), Path.open(), bz2.open(), etc. #3371

ilai-deutel · 2019-10-16T08:06:48Z

This PR adds overloaded signatures for {builtin,pathlib.Path,bz2,gwip,lzma}.open() functions, using Literal modes to determine the return types.

mypy tests pass after updating the open and Path.open plugins and making a few changes to the expected outputs: python/mypy#7794
Fixes #2911

Related discussion: python/mypy#7643

srittau · 2019-10-16T10:28:58Z

Thank you, that is very useful! Without doing a full review yet, I think it would make sense to consolidate the _OPEN_TEXT_MODE and _OPEN_BINARY_MODE constants in one file, at least where they are equal.

ilai-deutel · 2019-10-16T10:37:18Z

Do you have any suggestion on where to put these constants? When consolidating the constants in builtins.pyi and importing them in pathlib.pyi, everything seems to be working except the pytype tests.

Ran pytype with 1655 pyis, got 4 errors.
stdlib/3/pathlib.pyi: pytype.load_pytd.BadDependencyError: No _OPEN_TEXT_MODE in module __builtin__, referenced from 'pathlib'
stdlib/3/zipapp.pyi: pytype.load_pytd.BadDependencyError: No _OPEN_TEXT_MODE in module __builtin__, referenced from 'pathlib'
third_party/2/pathlib2.pyi: pytype.load_pytd.BadDependencyError: No _OPEN_TEXT_MODE in module __builtin__, referenced from 'pathlib2'
third_party/2and3/toml.pyi: pytype.load_pytd.BadDependencyError: No _OPEN_TEXT_MODE in module __builtin__, referenced from 'pathlib'

I fixed that issue in 0335f3f by defining these constants in pathlib.pyi as well instead of importing them, it is less ideal than keeping them in one place.

srittau · 2019-10-16T11:49:49Z

I'd probably put them into os/__init__.pyi, since while open is a builtin, conceptually it matches many of the function there. I have no real preference, though.

stdlib/2/__builtin__.pyi

ilai-deutel · 2019-10-16T15:18:43Z

@srittau Actually what about io, since io.open == open but os.open != open?

srittau · 2019-10-16T15:26:44Z

Good idea!

ilai-deutel · 2019-10-17T07:33:56Z

Moved to io

JelleZijlstra · 2019-10-25T03:21:49Z

There's some merge conflicts and a CI failure, could you take a look?

ilai-deutel · 2019-10-25T07:14:29Z

mypy_selftest fails because this PR changes the behavior of mypy.
For instance, reveal_type(open()) now produces error: All overload variants of "open" require at least one argument instead of Too few arguments for "open".

ilai-deutel/mypy@b5f8f01 fixes the mypy tests. I was waiting for this PR to be merged before opening one on the mypy repo, but I just went ahead and created it (python/mypy#7794) so that we can check that mypy tests pass on Travis CI.

JelleZijlstra · 2019-10-25T14:30:15Z

Thanks! We'll need to make simultaneous merges in both repos. I can do that in a few days.

Prepare for merging python/typeshed#3371 and the accompanying mypy PR.

stdlib/2and3/builtins.pyi

stdlib/2/__builtin__.pyi

bluetech

This is impressive work! I only checked the Python 3 builtins.open() stubs, they look correct to me.

I wonder if you considered making it return the actual types, instead of the IO protocol? Specifically, the open() documentation describes the return type as follows:

The type of file object returned by the open() function depends on the mode. When open() is used to open a file in a text mode ('w', 'r', 'wt', 'rt', etc.), it returns a subclass of io.TextIOBase (specifically io.TextIOWrapper). When used to open a file in a binary mode with buffering, the returned class is a subclass of io.BufferedIOBase. The exact class varies: in read binary mode, it returns an io.BufferedReader; in write binary and append binary modes, it returns an io.BufferedWriter, and in read/write mode, it returns an io.BufferedRandom. When buffering is disabled, the raw stream, a subclass of io.RawIOBase, io.FileIO, is returned.

By my count, encoding it fully would require 5(!) overloads, but for such an important function it might be worthwhile.

ilai-deutel · 2020-03-02T03:15:08Z

@bluetech io.BufferedReader, io.BufferedWriter, io.BufferedRandom and io.FileIO are not subclasses of BinaryIO, so I think that existing code that expects BinaryIO would fail.

For instance, mypy currently returns no error with the following code:

import io

from click.testing import EchoingStdin

f = open('a', 'rb')
g = open('b', 'wb')

EchoingStdin(input=f, output=g)

However, if f is a BufferedReader and g a BufferedWriter, mypy will complain:

error: Argument "input" to "EchoingStdin" has incompatible type "BufferedReader"; expected "BinaryIO"
error: Argument "output" to "EchoingStdin" has incompatible type "BufferedWriter"; expected "BinaryIO"

See also #1229

Perhaps these concrete classes should inherit from BinaryIO, there might a reason why they don't.

bluetech · 2020-03-02T09:22:43Z

@ilai-deutel Right.

Perhaps these concrete classes should inherit from BinaryIO, there might a reason why they don't.

I tried it. There are several issues, fixable except one: FileIO wants to inherit from both RawIOBase and BinaryIO. But RawIOBase can return None from read and write (if it is not possible to read/write in non-blocking mode), while BinaryIO doesn't allow that. This is actually a problem with current stubs -- can get None when type says it can't. One solution might be to just not have FileIO implement BinaryIO.

Anyway, this is probably already a separate discussion from this PR. Trying to switch to concrete types can be done after this PR is merged.

JelleZijlstra · 2020-05-28T02:40:58Z

Sorry for letting this slide! There's a merge conflict now.

…shed into ilai-deutel-overloaded-open

…oaded-open

python/typeshed#3371 causes this to produce a different error message. open() with no arguments doesn't seem like an important use case, so testing for it explicitly in pythoneval isn't all that useful.

JelleZijlstra · 2020-05-28T03:57:42Z

I fixed the merge conflict locally (https://github.com/JelleZijlstra/typeshed/tree/ilai-deutel-overloaded-open) but it looks like I don't have permission to push to the PR branch. I'll probably just open a separate PR from this branch. I also submitted python/mypy#8908 to remove the failing test from mypy, since it doesn't seem useful.

ilai-deutel · 2020-05-28T04:06:07Z

Sorry for letting this slide!

No worries!

I fixed the merge conflict locally (https://github.com/JelleZijlstra/typeshed/tree/ilai-deutel-overloaded-open) but it looks like I don't have permission to push to the PR branch. I'll probably just open a separate PR from this branch. I also submitted python/mypy#8908 to remove the failing test from mypy, since it doesn't seem useful.

You should now have write access to my branch, but feel free to open a separate PR with your existing branch if it's easier

JelleZijlstra · 2020-05-28T04:10:49Z

To github.com:ilai-deutel/typeshed.git
 ! [remote rejected]   ilai-deutel-overloaded-open -> overloaded-open (permission denied)
error: failed to push some refs to 'git@github.com:ilai-deutel/typeshed.git'

Didn't work unfortunately.

ilai-deutel · 2020-05-28T04:38:49Z

Looks like it's a GitHub issue, I get a javascript error (Uncaught QueryError) when I click the "Allow edits by maintainers" checkbox.

I pushed your merge commits to this branch

python/typeshed#3371 causes this to produce a different error message. open() with no arguments doesn't seem like an important use case, so testing for it explicitly in pythoneval isn't all that useful.

srittau

I think this is ready for the merge dance with mypy. I can't do it myself, unfortunately. A few remarks below, but those are not showstoppers.

stdlib/2and3/bz2.pyi

srittau · 2020-05-28T11:10:17Z

stdlib/2and3/bz2.pyi

+ encoding: Optional[str] = ...,
+ errors: Optional[str] = ...,
+ newline: Optional[str] = ...,
+ ) -> Union[BZ2File, TextIO]: ...


Union return types are usually problematic, since they force the use to assert or cast the return value, even if the user knows the return type. In this particular case I actually like the returned Union, since the last overload is only used if mode is dynamic, so type checking at runtime makes sence.

stdlib/3/gzip.pyi

stdlib/3/lzma.pyi

JelleZijlstra · 2020-05-28T13:41:58Z

CI is green now after I removed the mypy test, so merging this won't require any complications.

Co-authored-by: Sebastian Rittau <srittau@rittau.biz>

JelleZijlstra · 2020-05-28T16:20:39Z

🎉

…ython#3371) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Sebastian Rittau <srittau@rittau.biz>

ilai-deutel added 5 commits October 15, 2019 22:23

Overload on mode for open() and Path.open()

71506a8

Update Python2 stubs

6bec337

overload + Literal on BZ2, gzip, LZMA

23749a6

Apply black partially to builtin.pyi

952dc90

Fix pytype exceptions

0335f3f

JelleZijlstra reviewed Oct 16, 2019

View reviewed changes

stdlib/2/__builtin__.pyi Outdated Show resolved Hide resolved

Move _OpenBinaryMode and _OpenTextMode to io; refactor code

38e04fe

Fix missing import in io

1619c9b

Merge branch 'master' into overloaded-open

1eb282e

ilai-deutel mentioned this pull request Oct 25, 2019

Remove open plugin, not needed with overloaded signatures on typeshed python/mypy#7794

Closed

JelleZijlstra added a commit to JelleZijlstra/mypy that referenced this pull request Oct 30, 2019

sync typeshed

a3ba9c4

Prepare for merging python/typeshed#3371 and the accompanying mypy PR.

JelleZijlstra mentioned this pull request Oct 30, 2019

sync typeshed python/mypy#7819

Closed

JelleZijlstra added a commit to JelleZijlstra/mypy that referenced this pull request Oct 30, 2019

sync typeshed

6ea3b01

Prepare for merging python/typeshed#3371 and the accompanying mypy PR.

ilai-deutel added 3 commits November 10, 2019 23:34

Merge branch 'master' into overloaded-open

e6a4321

Merge branch 'master' into overloaded-open

3a631c0

Merge branch 'master' into overloaded-open

f924ef2

JukkaL reviewed Jan 15, 2020

View reviewed changes

stdlib/2and3/builtins.pyi Outdated Show resolved Hide resolved

JukkaL reviewed Jan 15, 2020

View reviewed changes

stdlib/2/__builtin__.pyi Outdated Show resolved Hide resolved

open() always return BinaryIO in Python2

7455811

ilai-deutel requested a review from JukkaL January 29, 2020 06:45

bluetech reviewed Feb 24, 2020

View reviewed changes

Merge branch 'master' into overloaded-open

529d3b8

ilai-deutel requested a review from JelleZijlstra April 18, 2020 00:03

JelleZijlstra added 2 commits May 27, 2020 20:26

Merge branch 'overloaded-open' of https://github.com/ilai-deutel/type…

a4f11f1

…shed into ilai-deutel-overloaded-open

Merge remote-tracking branch 'upstream/master' into ilai-deutel-overl…

70916ef

…oaded-open

JelleZijlstra mentioned this pull request May 28, 2020

remove open() pythoneval test python/mypy#8908

Merged

JelleZijlstra mentioned this pull request May 28, 2020

BinaryIO does not have peek method #3951

Closed

srittau mentioned this pull request May 28, 2020

typeshed types module #4131

Closed

srittau approved these changes May 28, 2020

View reviewed changes

srittau added the topic: io I/O related issues label May 28, 2020

Remove support for Python 3.4

7bc89fe

Co-authored-by: Sebastian Rittau <srittau@rittau.biz>

JelleZijlstra approved these changes May 28, 2020

View reviewed changes

JelleZijlstra merged commit 846d922 into python:master May 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More precise return types for open(), Path.open(), bz2.open(), etc. #3371

More precise return types for open(), Path.open(), bz2.open(), etc. #3371

ilai-deutel commented Oct 16, 2019 •

edited

Loading

srittau commented Oct 16, 2019

ilai-deutel commented Oct 16, 2019 •

edited

Loading

srittau commented Oct 16, 2019

ilai-deutel commented Oct 16, 2019

srittau commented Oct 16, 2019

ilai-deutel commented Oct 17, 2019

JelleZijlstra commented Oct 25, 2019

ilai-deutel commented Oct 25, 2019

JelleZijlstra commented Oct 25, 2019

bluetech left a comment

ilai-deutel commented Mar 2, 2020

bluetech commented Mar 2, 2020

JelleZijlstra commented May 28, 2020

JelleZijlstra commented May 28, 2020

ilai-deutel commented May 28, 2020

JelleZijlstra commented May 28, 2020

ilai-deutel commented May 28, 2020

srittau left a comment

srittau May 28, 2020

JelleZijlstra commented May 28, 2020

JelleZijlstra commented May 28, 2020

More precise return types for open(), Path.open(), bz2.open(), etc. #3371

More precise return types for open(), Path.open(), bz2.open(), etc. #3371

Conversation

ilai-deutel commented Oct 16, 2019 • edited Loading

srittau commented Oct 16, 2019

ilai-deutel commented Oct 16, 2019 • edited Loading

srittau commented Oct 16, 2019

ilai-deutel commented Oct 16, 2019

srittau commented Oct 16, 2019

ilai-deutel commented Oct 17, 2019

JelleZijlstra commented Oct 25, 2019

ilai-deutel commented Oct 25, 2019

JelleZijlstra commented Oct 25, 2019

bluetech left a comment

Choose a reason for hiding this comment

ilai-deutel commented Mar 2, 2020

bluetech commented Mar 2, 2020

JelleZijlstra commented May 28, 2020

JelleZijlstra commented May 28, 2020

ilai-deutel commented May 28, 2020

JelleZijlstra commented May 28, 2020

ilai-deutel commented May 28, 2020

srittau left a comment

Choose a reason for hiding this comment

srittau May 28, 2020

Choose a reason for hiding this comment

JelleZijlstra commented May 28, 2020

JelleZijlstra commented May 28, 2020

ilai-deutel commented Oct 16, 2019 •

edited

Loading

ilai-deutel commented Oct 16, 2019 •

edited

Loading