VS2017 running out of memory compiling ir_x86.c after recent AVX-512 additions

[derekbruening]

This happened with VS2013 too in the past: #3992. We split up some #included files to solve it then, but they have grown again.

The recent AVX-512 broadcast additions for #4534 in PR #4577 seem to have pushed it over the edge: but it seems non-deterministic since several prior builds on GA CI Server2016 VS2017 worked.

https://github.com/DynamoRIO/dynamorio/pull/4586/checks?check_run_id=1553077661

2020-12-14T21:09:44.3614062Z FAILED: suite/tests/CMakeFiles/api.ir.dir/api/ir_x86.c.obj 
2020-12-14T21:09:44.3616541Z C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\cl.exe  /nologo -DDR_HOST_ -DDR_REG_ENUM_COMPATIBILITY -DDYNAMORIO_STANDALONE -DX86_64 -I. -I..\suite\tests -I..\core\arch -Icmake\..\include -I..\core\drlibc -I..\core\lib -Iinclude\annotations -Isuite\tests\annotations -Iext\include -DSTANDALONE -DUSE_DYNAMO  /nologo /MP /GF /FS /GS- /MTd  /Zi /Od /W3 /WX -DNIGHTLY_REGRESSION -DNOT_DYNAMORIO_CORE /showIncludes /Fosuite\tests\CMakeFiles\api.ir.dir\api\ir_x86.c.obj /Fdsuite\tests\CMakeFiles\api.ir.dir\ /FS -c ..\suite\tests\api\ir_x86.c
2020-12-14T21:09:44.3618960Z d:\a\dynamorio\dynamorio\suite\tests\api\ir_x86.c(469) : fatal error C1002: compiler is out of heap space in pass 2
2020-12-14T21:09:44.3620066Z d:\a\dynamorio\dynamorio\suite\tests\api\ir_x86.c(458) : fatal error C1002: compiler is out of heap space in pass 2
2020-12-14T21:09:44.8225895Z ninja: build stopped: subcommand failed.

Compare these to the sizes listed in #3992:

$ wc suite/tests/api/ir*.h | sort -n
    65    453   3684 suite/tests/api/ir_x86_2args_avx512_vex.h
    68    506   4354 suite/tests/api/ir_x86_3args_avx512_vex.h
    69    427   4149 suite/tests/api/ir_x86_4args_avx512_evex.h
    97    459   5196 suite/tests/api/ir_x86_2args_avx512_evex_mask.h
   113    627   5553 suite/tests/api/ir_x86_all_opc.h
   193    876   6917 suite/tests/api/ir_x86_0args.h
   193    992   9481 suite/tests/api/ir_x86_1args.h
   217   1031  13206 suite/tests/api/ir_x86_3args_avx512_evex.h
   247   1279  15896 suite/tests/api/ir_x86_2args_avx512_evex.h
   327   1859  21705 suite/tests/api/ir_x86_2args_mm.h
   331   1405  17631 suite/tests/api/ir_x86_4args.h
   425   2055  26349 suite/tests/api/ir_x86_3args_avx.h
   460   2634  27811 suite/tests/api/ir_x86_2args.h
   485   2570  30857 suite/tests/api/ir_x86_3args.h
  1041   4823  72815 suite/tests/api/ir_x86_5args_avx512_evex_mask.h
  3247  12076 187105 suite/tests/api/ir_x86_3args_avx512_evex_mask.h
  3417  13754 217118 suite/tests/api/ir_x86_4args_avx512_evex_mask_A.h
  4105  16676 255355 suite/tests/api/ir_x86_4args_avx512_evex_mask_B.h
 15100  64502 925182 total

I think we need to split ir_x86_4args_avx512_evex_mask_B.h in half.

[derekbruening]

This happened in the next run too. Strangely the run right after merging in PR #4577 worked: https://github.com/DynamoRIO/dynamorio/pull/4586/checks?check_run_id=1552875348
But the next two failed, after removing the release build from the 64-bit job. In any case it is happening often enough that it is blocking #4131 and thus #4549. I will fix it since it is blocking me.

[derekbruening]

This just happened on master:

Github Actions CI workflow run FAILED!
Workflow: ci-windows/vs2017-64
Repository: DynamoRIO/dynamorio
Branch ref: refs/heads/master
SHA: b3bc4df
Triggering actor: derekbruening
Triggering event: push
Run Id: 453269191
See more details on github.com/DynamoRIO/dynamorio/actions/runs/453269191

debug-internal-64: **** 1 build errors ****
d:\a\dynamorio\dynamorio\suite\tests\api\ir_x86.c(489) : fatal error C1002: compiler is out of heap space in pass 2

So the split I did was not enough and there's still a non-deterministic compiler path that hits OOM?