Add signature for dataclasses.replace #14849

ikonst · 2023-03-07T04:47:26Z

Validate dataclassses.replace actual arguments to match the fields:

Unlike __init__, the arguments are always named.
All arguments are optional except for InitVars without a default value.

The tricks:

We're looking up type of the first positional argument ("obj") through private API. See Adjusting the signature of a function on the basis of caller types in plugin #10216, Let function signature hook have context of caller #14845.
We're preparing the signature of "replace" (for that specific dataclass) during the dataclass transformation and storing it in a "private" class attribute __mypy-replace (obviously not part of PEP-557 but contains a hyphen so should not conflict with any future valid identifier). Stashing the signature into the symbol table allows it to be passed across phases and cached across invocations. The stashed signature lacks the first argument, which we prepend at function signature hook time, since it depends on the type that replace is called on.

Based on #14526 but actually simpler.
Partially addresses #5152.

Remaining tasks

handle generic dataclasses
avoid data class transforms
fine-grained mode tests

ikonst · 2023-03-07T05:21:02Z

mypy/plugins/dataclasses.py

+    arg_names = [None]
+    arg_kinds = [ARG_POS]
+    arg_types = [obj_type]
+    for attr in dataclass["attributes"]:


I wish I could've deserialized metadata, or even accessed a list of DataclassAttributes that's already been deserialized...

Or maybe the right way to go is to replace dataclasses.replace with an OverloadedFuncDef that the transform (which has full context) will keep adding to.

The OverloadedFuncDef appears to be a non-starter since by the time I can do anything to replace dataclasses.replace with a synthesized OverloadedFuncDef, the call expression has been resolved with the pointer to the generic FunctionDef.

OK, solved it by creating a "secret" symbol with a signature at semantic analysis time, then using this signature in the function sig callback.

ikonst · 2023-03-08T16:37:34Z

@ilevkivskyi care to take a look?

ikonst · 2023-03-14T12:55:41Z

👋 @AlexWaygood

AlexWaygood

I'm just a triager over at mypy, so I'm afraid I don't have merge privileges! Here's a quick comment though:

AlexWaygood · 2023-03-14T13:11:13Z

test-data/unit/lib-stub/dataclasses.pyi

@@ -32,3 +32,5 @@ def field(*,


 class Field(Generic[_T]): pass
+
+def replace(obj: _T, **changes: Any) -> _T: ...


This is pretty different to typeshed's stub for replace. Would it be worth adding a pythoneval test as well, to check that this machinery all works with the full typeshed stubs for the stdlib, outside the context of mypy's test suite?

The stub has:

def replace(__obj: _DataclassT, **changes: Any) -> _DataclassT: ...

I can rename obj to __obj in this fixture. We can also change the stub to better reflect reality:

if sys.version_info >= (3, 9): def replace(obj: _DataclassT, /, **changes: Any) -> _DataclassT: ... else: def replace(obj: _DataclassT, **changes: Any) -> _DataclassT: ...

Can we just use the Python 3.9 syntax even for Python 3.8 targets? Passing obj as kw is deprecated in Python 3.8 so being a checker/linter we might as well just disallow it, no?

Do you think it matters much? For the plugin to function, replace has to:

exist

have a first argument

The **changes are optional, the retval is overwritten, and FWIW it doesn't look for a keyword arg obj since it's deprecated since Python 3.8 and was never a good idea to begin with, so we might as well just not support it, right?

We can also change the stub to better reflect reality:

No need: prepending the parameter name with __ is the py37-compatible way of declaring that a parameter is positional-only, as is described in PEP 484: https://peps.python.org/pep-0484/#positional-only-arguments. So the obj parameter is already marked as positional-only on all Python versions in typeshed's stubs.

I don't think you need to change anything in your existing fixtures/tests for this PR, I'm just suggesting adding a single additional test in the pythoneval.test file. Unlike mypy's other tests, the tests in that file don't use fixtures — they run with typeshed's full stdlib stubs. So it's a good sanity check to make sure that the machinery you've added works with typeshed's stubs for this function as well as with mypy's fixtures.

…konst/mypy into 2023-03-06-dataclasses-replace

test-data/unit/lib-stub/dataclasses.pyi

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

ilevkivskyi

This still looks like a hack, but OTOH it should not make things worse, so let's give it a try. But please be sure to at least fix the fine-grained test before merging.

ilevkivskyi · 2023-05-21T22:13:52Z

mypy/plugins/dataclasses.py

+        if replace_sym is None:
+            _fail_not_dataclass(ctx, display_typ, parent_typ)
+            return None
+        replace_sig = get_proper_type(replace_sym.type)


No need to call get+proper_type() here, since we know we added it, you can just add assert isinstance(..., ProperType) instead.

It ends up looking awkward like this:

-replace_sig = get_proper_type(replace_sym.type) +replace_sig = replace_sym.type +assert isinstance(replace_sig, ProperType) assert isinstance(replace_sig, CallableType)

Isn't it simpler to do what I did before, even if taking a few more cycles?

I think it actually looks fine to have two asserts. It has two additional benefits:

It clearly indicates the intent

It reduces number of get_proper_type() in the code, so people we hopefully cargo cult them less (we had a bunch of weird crashes caused by blindly copying some get_proper_type() calls).

ilevkivskyi · 2023-05-21T22:25:52Z

mypy/plugins/dataclasses.py

+        return None
+
+
+def _meet_replace_sigs(sigs: list[CallableType]) -> CallableType:


This looks like hack. Is this because the plugin systems expects a CallableType returned from the plugin? Maybe we can instead relax this expectation and allow returning say a union of callable types?

Ooh, that's an interesting idea. It didn't cross my mind I could return a union of callables and let mypy handle it in check_union_call.

p.s. I'll be happy to simplify this code, but do you reckon we should first change the plugin API and then rework this PR, or merge as-is, change plugin API and then refactor?

(A bit worried the devil's in the details and that maybe it's not as easy as it sounds, or not equivalent.)

I think it is better to do it in a follow up PR, but leave a TODO or open an issue (or both).

ilevkivskyi · 2023-05-21T22:26:21Z

mypy/plugins/dataclasses.py

+
+    assert isinstance(ctx.api, TypeChecker)
+    obj_type = ctx.api.expr_checker.accept(obj_arg)
+    # </hack>


This proposal still stands. Do we have an issue for this?

test-data/unit/check-dataclasses.test

test-data/unit/fine-grained-dataclass.test

ilevkivskyi · 2023-05-21T22:41:25Z

test-data/unit/fine-grained-dataclass.test

@@ -0,0 +1,25 @@
+[case replace]
+[file model.py]


I think this case may be not correct, there should be a top-level section like import replace. I am not sure what the test runner does when it is empty. (see testMetaclassOperators as a totally random example)

It's meaningful to import it? I would think mypy goes over all Python files, rather than follow the import chain.

e.g.

echo > foo.py echo "a: str = 42" > bar.py mypy .

results in

bar.py:1: error: Incompatible types in assignment (expression has type "int", variable has type "str") [assignment] F

I don't mind adding the import but it doesn't make the test any more/less meaningful (notice that it does produce the expected error).

I was worried fine-grained test runner doesn't simply check all files (like some other test runners), but I checked the code, and it actually looks OK.

…ses-replace

github-actions · 2023-05-22T02:35:07Z

Diff from mypy_primer, showing the effect of this PR on open source code:

vision (https://github.com/pytorch/vision) got 1.42x slower (29.2s -> 41.3s)

ikonst · 2023-06-02T14:10:25Z

@ilevkivskyi can take another look?

github-actions · 2023-06-02T14:35:05Z

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

github-actions · 2023-06-12T12:46:41Z

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

ikonst · 2023-06-13T00:05:26Z

@ilevkivskyi can take another look?

ilevkivskyi · 2023-06-16T22:55:59Z

@ikonst Please note there is now a merge conflict. You can also address couple remaining comments before I merge this.

…ses-replace

github-actions · 2023-06-17T03:56:04Z

Diff from mypy_primer, showing the effect of this PR on open source code:

graphql-core (https://github.com/graphql-python/graphql-core): typechecking got 1.06x slower (281.6s -> 297.4s)
(Performance measurements are based on a single noisy sample)

…5503) Now we use a similar approach to #14849 First, we generate a private name to store in a metadata (with `-`, so - no conflicts, ever). Next, we check override to be compatible: we take the currect signature and compare it to the ideal one we have. Simple and it works :) Closes #15498 Closes #9254 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ivan Levkivskyi <levkivskyi@gmail.com>

ikonst added 3 commits March 6, 2023 23:47

Add signature for dataclasses.replace

6dbaf9c

add ClassVar

4dcbe44

prevent misleading note

7ed3741

This comment has been minimized.

Sign in to view

ikonst commented Mar 7, 2023

View reviewed changes

ikonst added 3 commits March 7, 2023 23:40

stash in a secret class-private

89257b5

fix typing

32b1d47

docs and naming

1f08816

ikonst force-pushed the 2023-03-06-dataclasses-replace branch from f6c3a8f to 1f08816 Compare March 8, 2023 05:09

This comment has been minimized.

Sign in to view

ikonst added 2 commits March 8, 2023 11:00

inst -> obj

c456a5f

add the secret symbol to deps.test

8118d29

This comment has been minimized.

Sign in to view

ikonst added 3 commits March 8, 2023 13:34

language

789cb2b

nit

9f0974c

nit

a37e406

This comment has been minimized.

Sign in to view

Merge branch 'master' into 2023-03-06-dataclasses-replace

367c0e9

This comment has been minimized.

Sign in to view

AlexWaygood reviewed Mar 14, 2023

View reviewed changes

ikonst added 3 commits March 14, 2023 10:29

make obj positional-only

0e84c4f

Merge branch '2023-03-06-dataclasses-replace' of https://github.com/i…

9cfc081

…konst/mypy into 2023-03-06-dataclasses-replace

add pythoneval test

7b907cf

AlexWaygood reviewed Mar 14, 2023

View reviewed changes

test-data/unit/lib-stub/dataclasses.pyi Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

use py3.7 syntax

985db60

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

This comment has been minimized.

Sign in to view

Merge branch 'master' into 2023-03-06-dataclasses-replace

15dbb7b

ilevkivskyi approved these changes May 21, 2023

View reviewed changes

ikonst added 4 commits May 21, 2023 22:00

add to testReplaceUnion

8fe75a7

testReplaceUnion: fix B.y to be int

bea50e8

rename testcase replace to testReplace

cd35951

Merge remote-tracking branch 'origin/master' into 2023-03-06-dataclas…

d71a7c0

…ses-replace

Merge branch 'master' into 2023-03-06-dataclasses-replace

957744e

ikonst added 2 commits June 12, 2023 08:22

Merge remote-tracking branch 'origin/master' into pr/ikonst/14849

6abf6ff

remove get_expression_type hack

4c4fc94

ikonst mentioned this pull request Jun 17, 2023

Function signature hook returning a union #15457

Open

ikonst added 4 commits June 16, 2023 23:32

assert isinstance(replace_sig, ProperType)

be4a290

add a TODO for _meet_replace_sigs

ee0ae21

Merge remote-tracking branch 'origin/master' into 2023-03-06-dataclas…

3f656f8

…ses-replace

Merge remote-tracking branch 'origin/master' into 2023-03-06-dataclas…

65d6e89

…ses-replace

ilevkivskyi merged commit 6f2bfff into python:master Jun 17, 2023

ikonst deleted the 2023-03-06-dataclasses-replace branch June 17, 2023 13:16

jakkdl mentioned this pull request Jun 21, 2023

dataclasses.replace does not type correctly microsoft/pyright#1047

Closed

sobolevn mentioned this pull request Jun 23, 2023

Support better __post_init__ method signature for dataclasses #15503

Merged

tmke8 mentioned this pull request Jun 28, 2023

Typecheck dataclasses.replace semantically microsoft/pyright#5397

Closed

AlexWaygood mentioned this pull request Jul 7, 2023

Crash when iterating a Tuple list of @dataclass Protocol #15618

Closed

AlexWaygood mentioned this pull request Aug 11, 2023

Custom dataclasses are not recognized as dataclasses #15843

Closed

glatterf42 mentioned this pull request Aug 29, 2023

Updates to the water module, water avaialbility and country sub-annual features iiasa/message-ix-models#115

Merged

1 task

mcwitt mentioned this pull request Sep 25, 2023

Avoid unnecessary context creation in HREX, using BoundPotential::set_params proteneer/timemachine#1151

Merged

1 task

		@@ -32,3 +32,5 @@ def field(*,


		class Field(Generic[_T]): pass

		def replace(obj: _T, **changes: Any) -> _T: ...

		return None


		def _meet_replace_sigs(sigs: list[CallableType]) -> CallableType:

Add signature for dataclasses.replace #14849

Add signature for dataclasses.replace #14849

Conversation

ikonst commented Mar 7, 2023 • edited Loading

Remaining tasks

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

ikonst commented Mar 8, 2023

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ikonst commented Mar 14, 2023

AlexWaygood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexWaygood Mar 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ikonst May 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ikonst May 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 22, 2023

ikonst commented Jun 2, 2023

github-actions bot commented Jun 2, 2023

github-actions bot commented Jun 12, 2023

ikonst commented Jun 13, 2023

ilevkivskyi commented Jun 16, 2023

github-actions bot commented Jun 17, 2023

ikonst commented Mar 7, 2023 •

edited

Loading

AlexWaygood Mar 14, 2023 •

edited

Loading

ikonst May 22, 2023 •

edited

Loading

ikonst May 22, 2023 •

edited

Loading