Added more type system conformance tests (#1561)

Added tests for `@final` decorator.
Added tests for `Final` annotation.
Updated pyright test results for latest version (1.1.344).
Added tests for Annotated.
Added tests for @OverRide decorator.
Added tests for "Type checker directives" section of spec.
Added minimal tests for `@overload`.
Added tests for the "Special types in annotations" section of the typing spec.
Added conformance tests for the "Protocols" section of the type specification.
Added tests for PEP 695 type parameter functionality.
Improved error recovery and reporting for test tool if it cannot decode ".toml" file.
Added conformance tests for TypeVarTuple.
Added conformance tests for ParamSpec.
Added tests for "Variance" section of Generics chapter.
Expanded `@no_type_check` test to cover classes.
Implemented PR suggestions by @hauntsaninja.
Fixed bug in `generics_typevartuple_specialization` test. Added additional test cases in the `generics_variance` test.
Regenerated results page.
Enhanced protocols test to cover tuples of types passed as second argument to `isinstance` and `issubclass`.
Fixed a bug in the dataclasses_usage test. Fixed a couple of bugs in the generics_variance_inference test.
Fixed a bug in the protocols_definition test.
Added a new test condition in the protocols_runtime_checkable test.
Updated conformance results for pyright 1.1.345.
Updated look of conformance test results page so it's easier to read.
Made further refinements to conformance test results page.

conformance/results/mypy/dataclasses_usage.toml

@@ -12,5 +12,5 @@ dataclasses_usage.py:89: error: Incompatible types in assignment (expression has
 dataclasses_usage.py:127: error: Too many arguments for "DC7"  [call-arg]
 dataclasses_usage.py:130: error: Missing positional argument "y" in call to "DC8"  [call-arg]
 dataclasses_usage.py:179: error: Too many arguments for "DC13"  [call-arg]
-dataclasses_usage.py:207: error: Name "v1" already defined on line 25  [no-redef]
+dataclasses_usage.py:205: error: Name "v1" already defined on line 25  [no-redef]
 """

conformance/results/mypy/directives_no_type_check.toml

@@ -1,3 +1,7 @@
-conformant = "Pass"
+conformant = "Partial"
+notes = """
+Does not honor `@no_type_check` class decorator.
+"""
 output = """
+directives_no_type_check.py:16: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
 """

conformance/results/mypy/generics_paramspec_basic.toml

@@ -0,0 +1,17 @@
+conformant = "Partial"
+notes = """
+Does not reject ParamSpec when used "bare" in type alias definition.
+"""
+output = """
+generics_paramspec_basic.py:10: error: String argument 1 "NotIt" to ParamSpec(...) does not match variable name "WrongName"  [misc]
+generics_paramspec_basic.py:23: error: Invalid location for ParamSpec "P"  [valid-type]
+generics_paramspec_basic.py:23: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
+generics_paramspec_basic.py:27: error: Invalid location for Concatenate  [valid-type]
+generics_paramspec_basic.py:27: note: You can use Concatenate as the first argument to Callable
+generics_paramspec_basic.py:31: error: Invalid location for ParamSpec "P"  [misc]
+generics_paramspec_basic.py:31: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
+generics_paramspec_basic.py:35: error: Invalid location for ParamSpec "P"  [valid-type]
+generics_paramspec_basic.py:35: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
+generics_paramspec_basic.py:39: error: Invalid location for ParamSpec "P"  [valid-type]
+generics_paramspec_basic.py:39: note: You can use ParamSpec as the first argument to Callable, e.g., 'Callable[P, int]'
+"""

conformance/results/mypy/generics_paramspec_components.toml

@@ -0,0 +1,23 @@
+conformant = "Partial"
+notes = """
+Does not report illegal use of "P.args" on normal parameter.
+Does not report error when P.args is specified but P.kwargs is missing.
+Does not report error when P is out of scope and P.args and P.kwargs is used.
+Does not report error when keyword argument is specified between P.args and P.kwargs.
+Does not report error when calling callable and argument is missing for concatenated parameters.
+"""
+output = """
+generics_paramspec_components.py:17: error: Use "P.args" for variadic "*" parameter  [valid-type]
+generics_paramspec_components.py:17: error: Use "P.kwargs" for variadic "**" parameter  [valid-type]
+generics_paramspec_components.py:23: error: Use "P.kwargs" for variadic "**" parameter  [valid-type]
+generics_paramspec_components.py:49: error: Argument 1 has incompatible type "*P.kwargs"; expected "P.args"  [arg-type]
+generics_paramspec_components.py:49: error: Argument 2 has incompatible type "**P.args"; expected "P.kwargs"  [arg-type]
+generics_paramspec_components.py:51: error: Argument 1 has incompatible type "int"; expected "P.args"  [arg-type]
+generics_paramspec_components.py:70: error: Argument 1 has incompatible type "*P.args"; expected "int"  [arg-type]
+generics_paramspec_components.py:70: error: Argument 2 has incompatible type "int"; expected "P.args"  [arg-type]
+generics_paramspec_components.py:83: error: "foo" gets multiple values for keyword argument "x"  [misc]
+generics_paramspec_components.py:83: error: Argument 1 to "foo" has incompatible type "*P.args"; expected "int"  [arg-type]
+generics_paramspec_components.py:83: error: Argument 3 to "foo" has incompatible type "**P.kwargs"; expected "int"  [arg-type]
+generics_paramspec_components.py:98: error: Argument 2 to "twice" has incompatible type "str"; expected "int"  [arg-type]
+generics_paramspec_components.py:98: error: Argument 3 to "twice" has incompatible type "int"; expected "str"  [arg-type]
+"""

conformance/results/mypy/generics_paramspec_semantics.toml

@@ -0,0 +1,14 @@
+conformant = "Pass"
+output = """
+generics_paramspec_semantics.py:26: error: Unexpected keyword argument "a"  [call-arg]
+generics_paramspec_semantics.py:26: error: Unexpected keyword argument "b"  [call-arg]
+generics_paramspec_semantics.py:27: error: Argument 2 has incompatible type "str"; expected "bool"  [arg-type]
+generics_paramspec_semantics.py:61: error: Argument 2 to "func1" has incompatible type "Callable[[NamedArg(int, 'y')], int]"; expected "Callable[[NamedArg(int, 'x')], int]"  [arg-type]
+generics_paramspec_semantics.py:97: error: Argument 1 has incompatible type "int"; expected "str"  [arg-type]
+generics_paramspec_semantics.py:107: error: Argument 1 has incompatible type "int"; expected "bool"  [arg-type]
+generics_paramspec_semantics.py:119: error: Argument 1 has incompatible type "int"; expected "str"  [arg-type]
+generics_paramspec_semantics.py:126: error: Argument 1 to "expects_int_first" has incompatible type "Callable[[str], int]"; expected "Callable[[int], int]"  [arg-type]
+generics_paramspec_semantics.py:126: note: This is likely because "one" has named arguments: "x". Consider marking them positional-only
+generics_paramspec_semantics.py:131: error: Argument 1 to "expects_int_first" has incompatible type "Callable[[NamedArg(int, 'x')], int]"; expected "Callable[[int, NamedArg(int, 'x')], int]"  [arg-type]
+generics_paramspec_semantics.py:136: error: Argument 1 to "expects_int_first" has incompatible type "Callable[[KwArg(int)], int]"; expected "Callable[[int, KwArg(int)], int]"  [arg-type]
+"""

conformance/results/mypy/generics_paramspec_specialization.toml

@@ -0,0 +1,8 @@
+conformant = "Pass"
+output = """
+generics_paramspec_specialization.py:45: error: Can only replace ParamSpec with a parameter types list or another ParamSpec, got "int"  [misc]
+generics_paramspec_specialization.py:55: error: Argument 1 has incompatible type "str"; expected "int"  [arg-type]
+generics_paramspec_specialization.py:56: error: Argument 3 has incompatible type "str"; expected "bool"  [arg-type]
+generics_paramspec_specialization.py:61: error: Argument 1 has incompatible type "str"; expected "int"  [arg-type]
+generics_paramspec_specialization.py:62: error: Argument 3 has incompatible type "str"; expected "bool"  [arg-type]
+"""

conformance/results/mypy/generics_syntax_autovariance.toml

@@ -0,0 +1,2 @@
+output = """
+"""

conformance/results/mypy/generics_syntax_compatibility.toml

@@ -0,0 +1,14 @@
+conformant = "Unsupported"
+notes = """
+Type parameter syntax not yet support.
+"""
+output = """
+generics_syntax_compatibility.py:14: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_compatibility.py:14: error: Name "V" is not defined  [name-defined]
+generics_syntax_compatibility.py:18: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_compatibility.py:18: error: Name "V" is not defined  [name-defined]
+generics_syntax_compatibility.py:22: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_compatibility.py:23: error: Name "V" is not defined  [name-defined]
+generics_syntax_compatibility.py:26: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_compatibility.py:26: error: Name "M" is not defined  [name-defined]
+"""

conformance/results/mypy/generics_syntax_declarations.toml

@@ -0,0 +1,35 @@
+conformant = "Unsupported"
+notes = """
+Type parameter syntax not yet support.
+"""
+output = """
+generics_syntax_declarations.py:13: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:14: error: Expression is of type "Any", not "TypeVar"  [assert-type]
+generics_syntax_declarations.py:14: error: Name "T" is not defined  [name-defined]
+generics_syntax_declarations.py:15: error: Expression is of type "Any", not "TypeVarTuple"  [assert-type]
+generics_syntax_declarations.py:15: error: Name "Ts" is not defined  [name-defined]
+generics_syntax_declarations.py:16: error: Expression is of type "Any", not "ParamSpec"  [assert-type]
+generics_syntax_declarations.py:16: error: Name "P" is not defined  [name-defined]
+generics_syntax_declarations.py:19: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:19: error: Free type variable expected in Generic[...]  [misc]
+generics_syntax_declarations.py:19: error: Name "T" is not defined  [name-defined]
+generics_syntax_declarations.py:23: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:27: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:27: error: Free type variable expected in Protocol[...]  [misc]
+generics_syntax_declarations.py:27: error: Name "S" is not defined  [name-defined]
+generics_syntax_declarations.py:27: error: Name "T" is not defined  [name-defined]
+generics_syntax_declarations.py:31: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:32: error: Name "T" is not defined  [name-defined]
+generics_syntax_declarations.py:37: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:41: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:45: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:46: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:50: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:54: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:58: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:62: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:66: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:73: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:77: error: PEP 695 generics are not yet supported  [valid-type]
+generics_syntax_declarations.py:81: error: PEP 695 generics are not yet supported  [valid-type]
+"""