Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using typing_extension.Unpack results in INTERNAL ERROR #13981

Closed
stoney95 opened this issue Nov 1, 2022 · 2 comments · Fixed by #15924
Closed

Using typing_extension.Unpack results in INTERNAL ERROR #13981

stoney95 opened this issue Nov 1, 2022 · 2 comments · Fixed by #15924
Labels
crash topic-pep-646 PEP 646 (TypeVarTuple, Unpack)

Comments

@stoney95
Copy link

stoney95 commented Nov 1, 2022

Bug Report
I'm trying to use typing_extensions.Unpack in python 3.7.11 to check the annotation of a higher order function. The function with annotation looks like this:

from typing import Tuple, TypeVar
from typing_extension import ParamSpec, Unpack

pipeline(*funcs: Unpack[Tuple[Callable[P, Any], Unpack[Tuple[Callable, ...]], Callable[..., Output]]]) -> Callable[P, Output]:
  ...

I first used the latest published version of mypy (0.982) to do the type checking. this lead to this error:

test_unpack.py:37: error: "Unpack" is not supported yet, use --enable-incomplete-features

Running with the suggested flag --enable-incomplete-features produced another error:

test_unpack.py:37: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
If this issue continues with mypy master, please report a bug at https://github.com/python/mypy/issues
version: 0.982
delete_me.py:37: : note: please use --show-traceback to print a traceback when reporting a bug

So I installed mypy from source. The build version is: 1.0.0+dev.55d0adf17a15ddbd0a8e9fb9d27f848117522a17.
I ran the type check again with the Unpack feature enabled: mypy test_unpack.py --enable-incomplete-feature=Unpack. This resulted in the same error:

test_unpack.py:37: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.0.0+dev.55d0adf17a15ddbd0a8e9fb9d27f848117522a17
Traceback (most recent call last):
  File "/Users/simons/anaconda3/envs/pypely/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/main.py", line 95, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/main.py", line 174, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/build.py", line 194, in build
    sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr, extra_plugins
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/build.py", line 276, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/build.py", line 2903, in dispatch
    process_graph(graph, manager)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/build.py", line 3287, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/build.py", line 3388, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/build.py", line 2309, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 465, in check_first_pass
    self.accept(d)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 573, in accept
    stmt.accept(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/nodes.py", line 822, in accept
    return visitor.visit_func_def(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 938, in visit_func_def
    self._visit_func_def(defn)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 942, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 1011, in check_func_item
    self.check_func_def(defn, typ, name, allow_empty)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 1208, in check_func_def
    self.accept(item.body)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 573, in accept
    stmt.accept(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/nodes.py", line 1218, in accept
    return visitor.visit_block(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 2569, in visit_block
    self.accept(s)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 573, in accept
    stmt.accept(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/nodes.py", line 1301, in accept
    return visitor.visit_assignment_stmt(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 2617, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checker.py", line 2840, in check_assignment
    rvalue_type = self.expr_checker.accept(rvalue, type_context=type_context)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 4665, in accept
    typ = node.accept(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/nodes.py", line 1838, in accept
    return visitor.visit_call_expr(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 410, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 531, in visit_call_expr_inner
    callee_type, e, fullname, object_type, member
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 1189, in check_call_expr_with_callee_type
    object_type=object_type,
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 1276, in check_call
    callee, args, arg_kinds, arg_names, callable_name, object_type, context
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 2142, in check_overload_call
    context,
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 2295, in infer_overload_return_type
    object_type=object_type,
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 1272, in check_call
    object_type,
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 1407, in check_callable_call
    callee, args, arg_kinds, formal_to_actual, context
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 1732, in infer_function_type_arguments
    callee_type, args, arg_kinds, formal_to_actual, inferred_args, context
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/checkexpr.py", line 1793, in infer_function_type_arguments_pass2
    context=self.argument_infer_context(),
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/infer.py", line 55, in infer_function_type_arguments
    callee_type, arg_types, arg_kinds, formal_to_actual, context
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/constraints.py", line 147, in infer_constraints_for_callable
    c = infer_constraints(callee.arg_types[i], actual_type, SUPERTYPE_OF)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/constraints.py", line 189, in infer_constraints
    return _infer_constraints(template, actual, direction)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/constraints.py", line 268, in _infer_constraints
    return template.accept(ConstraintBuilderVisitor(actual, direction))
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/types.py", line 1792, in accept
    return visitor.visit_callable_type(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/constraints.py", line 917, in visit_callable_type
    res.extend(infer_constraints(t, a, neg_op(self.direction)))
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/constraints.py", line 189, in infer_constraints
    return _infer_constraints(template, actual, direction)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/constraints.py", line 268, in _infer_constraints
    return template.accept(ConstraintBuilderVisitor(actual, direction))
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/types.py", line 933, in accept
    return visitor.visit_unpack_type(self)
  File "/Users/simons/anaconda3/envs/pypely/lib/python3.7/site-packages/mypy/constraints.py", line 540, in visit_unpack_type
    raise NotImplementedError
NotImplementedError: 
test_unpack.py:37: : note: use --pdb to drop into pdb

To Reproduce

from typing import Any, Callable, Tuple, TypeVar
from typing_extensions import ParamSpec, Unpack
from functools import reduce

P = ParamSpec("P")
CombineFirstOutput = TypeVar("CombineFirstOutput")
CombineSecondOutput = TypeVar("CombineSecondOutput")
Output = TypeVar("Output")

def double(x: int) -> Tuple[int, int]:
    return x*2, x

def add(x: int, y: int) -> int:
    return x + y

def to_string(x: int) -> str:
    return str(x)

def combine(
    func1: Callable[P, CombineFirstOutput], 
    func2: Callable[[CombineFirstOutput], CombineSecondOutput]
) -> Callable[P, CombineSecondOutput]:
    def _inner(*args: P.args, **kwargs: P.kwargs) -> CombineSecondOutput:
        return func2(func1(*args, **kwargs))

    return _inner


def pipeline(*funcs: Unpack[Tuple[Callable[P, Any], Unpack[Tuple[Callable, ...]], Callable[..., Output]]]) -> Callable[P, Output]:
    func = reduce(combine, funcs)

    return func

test2 = pipeline(add, double, add, to_string)

result = test2(3,4)
print(result.upper())

Expected Behavior
A passing type check

Actual Behavior
INTERNAL ERROR occured, which is asking to raise an issue here. As the last error in the verbose traceback is a NotImplementedError I guess the expected behaviour is not implemented yet. This issue is raised as it's stated in the error message.

Your Environment

  • Mypy version used: 1.0.0+dev.55d0adf17a15ddbd0a8e9fb9d27f848117522a17 (compiled: no)
  • Mypy command-line flags: --enable-incomplete-feature=Unpack
  • Mypy configuration options from mypy.ini (and other config files): -
  • Python version used: 3.7.11
@stoney95 stoney95 added the bug mypy got something wrong label Nov 1, 2022
@AlexWaygood AlexWaygood added crash topic-pep-646 PEP 646 (TypeVarTuple, Unpack) and removed bug mypy got something wrong labels Nov 1, 2022
@ilevkivskyi
Copy link
Member

cc @jhance FYI

@hauntsaninja hauntsaninja mentioned this issue Dec 19, 2022
17 tasks
@hauntsaninja
Copy link
Collaborator

Similarly:

from typing import Any, TypeVar
from typing_extensions import Unpack
T = TypeVar("T")

def convert(obj: Any, *to_classes: Unpack[tuple[type[T], ...]]) -> T | None:
    for class_ in to_classes:
        try:
            return class_(...)
        except:
            pass

x = convert(1, int, str)
reveal_type(x)  # int | str | None

ilevkivskyi added a commit that referenced this issue Aug 23, 2023
Fixes #13981
Fixes #15241
Fixes #15495
Fixes #15633
Fixes #15667
Fixes #15897
Fixes #15929

OK, I started following the plan outlined in
#15879. In this PR I focused mostly
on "kinematics". Here are some notes (in random order):
* I decided to normalize `TupleType` and `Instance` items in
`semanal_typeargs.py` (not in the type constructors, like for unions).
It looks like a simpler way to normalize for now. After this, we can
rely on the fact that only non-trivial (more below on what is trivial)
variadic items in a type list is either `*Ts` or `*tuple[X, ...]`. A
single top-level `TupleType` can appear in an unpack only as type of
`*args`.
* Callables turned out to be tricky. There is certain tight coupling
between `FuncDef.type` and `FuncDef.arguments` that makes it hard to
normalize prefix to be expressed as individual arguments _at
definition_. I faced exactly the same problem when I implemented `**`
unpacking for TypedDicts. So we have two choices: either handle prefixes
everywhere, or use normalization helper in relevant code. I propose to
go with the latter (it worked well for `**` unpacking).
* I decided to switch `Unpack` to be disallowed by default in
`typeanal.py`, only very specific potions are allowed now. Although this
required plumbing `allow_unpack` all the way from `semanal.py`,
conceptually it is simple. This is similar to how `ParamSpec` is
handled.
* This PR fixes all currently open crash issues (some intentionally,
some accidentally) plus a bunch of TODOs I found in the tests (but not
all).
* I decided to simplify `TypeAliasExpr` (and made it simple reference to
the `SymbolNode`, like e.g. `TypedDictExpr` and `NamedTupleExpr`). This
is not strictly necessary for this PR, but it makes some parts of it a
bit simpler, and I wanted to do it for long time.

Here is a more detailed plan of what I am leaving for future PRs (in
rough order of priority):
* Close non-crash open issues (it looks like there are only three, and
all seem to be straightforward)
* Handle trivial items in `UnpackType` gracefully. These are `<nothing>`
and `Any` (and also potentially `object`). They can appear e.g. after a
user error. Currently they can cause assert crashes. (Not sure what is
the best way to do this).
* Go over current places where `Unpack` is handled, and verify both
possible variadic items are handled.
* Audit variadic `Instance` constrains and subtyping (the latter is
probably OK, but the former may be broken).
* Audit `Callable` and `Tuple` subtyping for variadic-related edge cases
(constraints seem OK for these).
* Figure out story about `map_instance_to_supertype()` (if no changes
are needed, add tests for subclassing).
* Clear most remaining TODOs.
* Go once more over the large scale picture and check whether we have
some important parts missing (or unhandled interactions between those).
* Verify various "advanced" typing features work well with
`TypeVarTuple`s (and add some support if missing but looks important).
* Enable this feature by default.

I hope to finish these in next few weeks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
crash topic-pep-646 PEP 646 (TypeVarTuple, Unpack)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants