Windows CI builds broken

[JukkaL]

Currently all Windows CI builds seem to be broken.

Example failure: https://github.com/python/mypy/actions/runs/2044190861

This might be caused by the switch to a new CI image with Windows Server 2022: actions/runner-images#4856

We could try falling back to Windows-2019 temporarily while we investigate what is going on.

Any help with investigating the failures would be appreciated, since I don't do much development in Windows.

[JelleZijlstra]

The only suspicious thing I saw in the output is cl : Command line warning D9002 : ignoring unknown option '/O0'. https://docs.microsoft.com/en-us/cpp/build/reference/o-options-optimize-code?view=msvc-170 doesn't mention O0. Maybe it's a setuptools issue?

[emmatyping]

I can take a look at the issue a little later.

[KotlinIsland]

last pass was https://github.com/python/mypy/runs/5694572863?check_suite_focus=true
first fail was https://github.com/python/mypy/runs/5700430887?check_suite_focus=true

The last pass was on windows server 2022, so I don't think that is the root cause.

Sample failed output

__________________________________ testAsync __________________________________
[gw0] win32 -- Python 3.7.9 D:\a\mypy\mypy\.tox\py37\Scripts\python.EXE
data: D:\a\mypy\mypy\mypyc\test-data\run-misc.test:3:
D:\a\mypy\mypy\mypyc\test\test_run.py:137: in run_case
    self.run_case_inner(testcase)
D:\a\mypy\mypy\mypyc\test\test_run.py:153: in run_case_inner
    self.run_case_step(testcase, step)
D:\a\mypy\mypy\mypyc\test\test_run.py:266: in run_case_step
    assert glob.glob('native.*.{}'.format(suffix))
E   AssertionError: assert []
E    +  where [] = <function glob at 0x041B36A8>('native.*.pyd')
E    +    where <function glob at 0x041B36A8> = glob.glob
E    +    and   'native.*.pyd' = <built-in method format of str object at 0x05D9D1B0>('pyd')
E    +      where <built-in method format of str object at 0x05D9D1B0> = 'native.*.{}'.format
---------------------------- Captured stdout call -----------------------------
running build_ext
building 'native' extension
creating build\temp.win32-3.7
creating build\temp.win32-3.7\Release
creating build\temp.win32-3.7\Release\build
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\bin\HostX86\x86\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\a\mypy\mypy\mypyc\lib-rt -ID:\a\mypy\mypy\.tox\py37\include -Ic:\hostedtoolcache\windows\python\3.7.9\x86\include -Ic:\hostedtoolcache\windows\python\3.7.9\x86\Include "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" /Tcbuild\__native.c /Fobuild\temp.win32-3.7\Release\build\__native.obj /O0 /wd4102 /wd4101 /wd4146
__native.c

creating C:\Users\RUNNER~1\AppData\Local\Temp\mypy-test-wpgouzfj\tmp\build\lib.win32-3.7
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\bin\HostX86\x86\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:D:\a\mypy\mypy\.tox\py37\libs /LIBPATH:c:\hostedtoolcache\windows\python\3.7.9\x86\libs /LIBPATH:c:\hostedtoolcache\windows\python\3.7.9\x86 /LIBPATH:D:\a\mypy\mypy\.tox\py37\PCbuild\win32 "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\ATLMFC\lib\x86" "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.31.31103\lib\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x86" /EXPORT:PyInit_native build\temp.win32-3.7\Release\build\__native.obj /OUT:build\lib.win32-3.7\native.pyd /IMPLIB:build\temp.win32-3.7\Release\build\native.lib
   Creating library build\temp.win32-3.7\Release\build\native.lib and object build\temp.win32-3.7\Release\build\native.exp

Generating code

Finished generating code

copying build\lib.win32-3.7\native.pyd -> 
---------------------------- Captured stderr call -----------------------------
cl : Command line warning D9002 : ignoring unknown option '/O0'

Here are the details from "Set up job":

Successful job

Current runner version: '2.289.1'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Virtual Environment
  Environment: windows-2022
  Version: 20220316.1
  Included Software: https://github.com/actions/virtual-environments/blob/win22/20220316.1/images/win/Windows2022-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win22%2F20220316.1
Virtual Environment Provisioner
  1.0.0.0-main-20220307-1

Failed job

Current runner version: '2.289.1'
Operating System
  Microsoft Windows Server 2022
  10.0.20348
  Datacenter
Virtual Environment
  Environment: windows-2022
  Version: 20220316.1
  Included Software: https://github.com/actions/virtual-environments/blob/win22/20220316.1/images/win/Windows2022-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win22%2F20220316.1
Virtual Environment Provisioner
  1.0.0.0-main-20220307-1

Additional analysis

unknown option '/O0'`

There is a warning: cl : Command line warning D9002 : ignoring unknown option '/O0'

/O0 is being passed, which is the MYPYC_OPT_LEVEL being set here:

mypy/mypyc/test/test_run.py

Line 246 in 1480344

opt_level = int(os.environ.get('MYPYC_OPT_LEVEL', 0))

And being used here:

mypy/mypyc/build.py

Lines 525 to 529 in 1480344

	elif compiler.compiler_type == 'msvc':
	if opt_level == '3':
	opt_level = '2'
	cflags += [
	'/O{}'.format(opt_level),

#12462 Changed build to not pass the argument at all if the opt level was 0, which removed the warning but didn't resolve the failures, I still this this is a good change though and have re-raised the change here #12468, slightly modified to map 0 to 1

[JelleZijlstra]

In #12462 I tried to fix the /O0 issue; unfortunately it didn't fix the build.

[pranavrajpal]

setuptools had some releases around the time CI started failing, and since the bug seems to be related to the names of the generated extension modules and I think we use setuptools to handle the compilation, a setuptools change could cause a failure there.

The last pass in #12460 (comment) seems to be using setuptools 61.0.0, and the first fail seems to be using 61.1.0.

I haven't looked too much into what the cause of the failures is, but it seems worth trying to pin setuptools to 61.0.0 for now to see if that fixes the CI.

[emmatyping]

I started looking into it last night, but it seems that the test suite hangs and I cannot Ctrl+C when it fails for me, which is unfortunate. I will try to run things with failfast and see if using an older setuptools fixes things. (Hopefully it fixes the test hanging as well, but that may be another issue).

[jhance]

https://github.com/python/mypy/runs/5714129295?check_suite_focus=true

Doesn't seem like its setuptools?