stubtest: fix literal type construction

[sobolevn]

I've started with a new unit test that illustrates the problem.

Closes python/typeshed#6845

[sobolevn]

Failure!

 =================================== FAILURES ===================================
__________________________ StubtestUnit.test_literal ___________________________
[gw1] linux -- Python 3.10.1 /home/runner/work/mypy/mypy/.tox/py/bin/python

args = (<mypy.test.teststubtest.StubtestUnit testMethod=test_literal>,)
kwargs = {}, cases = [<mypy.test.teststubtest.Case object at 0x7f33ebc511b0>]
expected_errors = set()
c = <mypy.test.teststubtest.Case object at 0x7f33ebc511b0>, @py_assert1 = False
@py_format3 = "{'test_module..._module.BYT2'} == set()\n~Extra items in the left set:\n~'test_module.BYT1'\n~'test_module.BYT2'\n~Use -v to get the full diff"
@py_format5 = "test_module.BYT1\n~test_module.BYT2\n~\n>assert {'test_module..._module.BYT2'} == set()\n~Extra items in the left set:\n~'test_module.BYT1'\n~'test_module.BYT2'\n~Use -v to get the full diff"
output = 'test_module.BYT1\ntest_module.BYT2\n'
actual_errors = {'test_module.BYT1', 'test_module.BYT2'}

    def test(*args: Any, **kwargs: Any) -> None:
        cases = list(fn(*args, **kwargs))
        expected_errors = set()
        for c in cases:
            if c.error is None:
                continue
            expected_error = "{}.{}".format(TEST_MODULE_NAME, c.error)
            assert expected_error not in expected_errors, (
                "collect_cases merges cases into a single stubtest invocation; we already "
                "expect an error for {}".format(expected_error)
            )
            expected_errors.add(expected_error)
        output = run_stubtest(
            stub="\n\n".join(textwrap.dedent(c.stub.lstrip("\n")) for c in cases),
            runtime="\n\n".join(textwrap.dedent(c.runtime.lstrip("\n")) for c in cases),
            options=["--generate-allowlist"],
        )
    
        actual_errors = set(output.splitlines())
>       assert actual_errors == expected_errors, output
E       AssertionError: test_module.BYT1
E         test_module.BYT2
E         
E       assert {'test_module..._module.BYT2'} == set()
E         Extra items in the left set:
E         'test_module.BYT1'
E         'test_module.BYT2'
E         Use -v to get the full diff

mypy/test/teststubtest.py:128: AssertionError
=========================== short test summary info ============================
FAILED mypy/test/teststubtest.py::StubtestUnit::test_literal - AssertionError...
====== 1 failed, 9679 passed, 371 skipped, 7 xfailed in 684.78s (0:11:24) ======

Link: https://github.com/python/mypy/runs/4736271970?check_suite_focus=true

Now I will try to fix it.

[sobolevn]

Done! CC @hauntsaninja

[sobolevn]

Btw, I am almost sure that it does not work anymore, because bool() now is Union[Literal[False], Literal[True]].

I can address this in a new PR.

[sobolevn]

Looks like mypyc-compiled mypy fails. Here's the reason why: https://github.com/python/mypy/blame/e8cf960e8579e5cee2db27212efad4870e5106bd/mypy/stubtest.py#L1032-L1041

Any ideas?

[hauntsaninja]

Looking, I can push a fix to your branch!

[sobolevn]

Let's wait for my new solution to build. Maybe I've found a good workaround. But, mypyc builds need to finish.

I can push a fix to your branch!

Yes, Allow edits and access to secrets by maintainers is enabled.

[hauntsaninja]

Oh just saw your message. Our fixes are quite similar. We should be able to change enum handling as well and get rid of that hacky try catch completely (what I pushed isn't quite right). I also collate the tests... it makes running stubtest tests much faster (since each test method involves writing files to disk)

[sobolevn]

LGTM!

[hauntsaninja]

As a result, this should be fixed as well https://github.com/python/typeshed/blob/032e6ee90cbb938259a2aa2183966502de19108e/tests/stubtest_allowlists/py3_common.txt#L108 :-)

[sobolevn]

Awesome! I will send a PR to fix it, when new mypy will be out!