Fix 4312 and add Windows on ARM64 support for msvc. #4324

jcbrill · 2023-03-14T15:41:25Z

Changes for #4312:

The installed VCS list is cached but has an indirect dependency on the value of the environment's TARGET_ARCH during construction. For the initial construction, force the env['TARGET_ARCH'] to be undefined and then restore the value after construction.
Protect against an empty regular expression list when iteratively removing default tool sequences.

Changes for ARM64 host support:

Add configuration data structures to support ARM64 hosts and ARM64 native tools for VS2022. Update the tests for the renamed data structures.
Evaluate the PROCESSOR_ARCHITECTURE value from the windows registry before evaluating the os environment values for host architecture determination on Windows.
Set VSCMD_SKIP_SENDTELEMETRY=1 for arm32 process on arm64 host if not already defined to prevent powershell dll not found error.
The os environment variable value for PROCESSOR_ARCHITECTURE is AMD64 for Windows ARM64 hosts when using a python built for AMD64.

Related MSVC changes for #4312 and ARM64 hosts:

Check that cl.exe exists in find_batch_file for VS6 to VS2015.
Move the sdk batch file query code to its own function. Query for the sdk batch file only when all of the vc script host/target combinations have been exhausted and a valid script has not been found.
Hoist the query for the vc product directory outside the current vc script host/target loop. Catch the internal exceptions as before.
Clear the sdk batch file path for VCForPython as the sdk batch files do not appear to be updated during installation and do not point to the VCForPython installation location.
Always check that cl.exe is found in vc script environment path.

Miscellaneous:

Reorder and group exception definitions in Tool\MSCommon\vc.py by external exceptions and internal exceptions.
Adjust debug messages.
Convert the floating point msvc version to an integer version number for product range comparisons (e.g., 14.3 is 143). Adjust the comparison ranges accordingly.
Refactor MSCommon\common.py logging filter class due to SCons\SConfTests.py failures after module reset when debug logging is enabled.
Add arm64 to the MSVS supported architectures list for VS2017 and later to be consistent with the current documentation of MSVS_ARCH.

Known issues;

Test test\Configure\config-h.py fails on non-AMD64 hosts with the default VS2015+ target architecture. The printf function is exported only from the AMD64 libraries used for the test. This test fails on both x86 and ARM64 hosts with VS2015 and later installed.
See MSVC: Configure test failure on non-AMD64 hosts with default VS 2015+ target architecture #4320 for details.

Contributor Checklist:

I have created a new test or updated the unit tests to cover the new/changed functionality.
I have updated CHANGES.txt (and read the README.rst)
I have updated RELEASE.txt
I have updated the appropriate documentation

Changes for SCons#4312: * The installed VCS list is cached but has an indirect dependency on the value of the environment's TARGET_ARCH during construction. For the initial construction, force the env['TARGET_ARCH'] to be undefined and then restore the value after construction. * Protect against an empty regular expression list when iteratively removing default tool sequences. Changes for ARM64 host support: * Add configuration data structures to support ARM64 hosts and ARM64 native tools for VS2022. Update the tests for the renamed data structures. * Evaluate the PROCESSOR_ARCHITECTURE value from the windows registry before evaluating the os environment values for host architecture determination on Windows. * Set VSCMD_SKIP_SENDTELEMETRY=1 for arm32 process on arm64 host if not already defined to prevent powershell dll not found error. * The os environment variable value for PROCESSOR_ARCHITECTURE is AMD64 for Windows ARM64 hosts when using a python built for AMD64. Related MSVC changes for SCons#4312 and ARM64 hosts: * Check that cl.exe exists in find_batch_file for VS6 to VS2015. Move the sdk batch file query code to its own function. Query for the sdk batch file only when all of the vc script host/target combinations have been exhausted and a valid script has not been found. * Hoist the query for the vc product directory outside the current vc script host/target loop. Catch the internal exceptions as before. * Clear the sdk batch file path for VCForPython as the sdk batch files do not appear to be updated during installation and do not point to the VCForPython installation location. * Move the sdk batch file determination to its own function. Defer evaluation of candidate sdk batch files until after all host/target combinations of the vc scripts have been evaluated. * Always check that cl.exe is found in vc script environment path. Miscellaneous: * Reorder and group exception definitions in Tool\MSCommon\vc.py by external exceptions and internal exceptions. * Adjust debug messages. * Convert the floating point msvc version to an integer version number for product range comparisons (e.g., 14.3 is 143). Adjust the comparison ranges accordingly.

…dule reset when debug logging is enabled.

jcbrill · 2023-03-18T15:13:31Z

Test Environments

Test Env 1 - Windows 10 Pro on AMD64 [AMD Ryzen Threadripper 3960X, 64GB]

Visual Studio installations:
- 14.3 2022 Community
- 14.2 2019 Community
- 14.1 2017 Community
- 14.1Exp 2017 Express
- 14.0 2015 Community
- 12.0 2013 Community
- 11.0Exp 2012 Express
- 10.0Exp 2010 Express
- 9.0 2008 VCForPython
- 9.0 2008 Professional
- 8.0 2005 Professional
- 7.1 2003 Professional
- 7.0 2002 Professional
- 6.0 Professional
Stress tests:
- Configuration:
  - invoked on local drive
  - msvc debug log enabled
- Combinations: 1137:
  - Python 3.6.8 x86 embeddable
  - Python 3.6.8 amd64 embeddable
  - Python 3.11.0a5 x86 embeddable
  - Python 3.11.0a5 amd64 embeddable
runtest.py --all [1271 tests]:
- Configuration:
  - invoked on local drive
  - msvc cache enabled
  - msvc debug log enabled
- WinPython 3.6.8 x86 venv [6 failed, 251 no result]
- WinPython 3.6.8 amd64 venv [6 failed, 244 no result]
- WinPython 3.10.9 x86 venv [6 failed, 251 no result]
- WinPython 3.10.9 amd64 venv [6 failed, 251 no result]
- Common failures [6 failed]:
  - ninja [1 failed]:
    - test\ninja\iterative_speedup.py
      Ninja was slower than SCons: SCons: 4.173s Ninja: 24.733s
      Ninja was slower than SCons: SCons: 4.892s Ninja: 24.570s
      Ninja was slower than SCons: SCons: 4.119s Ninja: 24.314s
      Ninja was slower than SCons: SCons: 5.727s Ninja: 25.872s
  - virtualenv [5 failed]:
    - test\virtualenv\activated\option\enable-virtualenv.py
    - test\virtualenv\activated\option\ignore-virtualenv.py
    - test\virtualenv\activated\virtualenv_activated_python.py
    - test\virtualenv\activated\virtualenv_detect_virtualenv.py
    - test\virtualenv\unactivated\virtualenv_unactivated_python.py
    - local installation: system (C:), venv (S:), python (E:), source(S:)
      ValueError: path is on mount 'C:', start on mount 'S:'

Test Env 2 - Windows 11 Pro on ARM64 [Snapdragon 7c, 8GB]

Visual Studio installations:
- 14.3 2022 Community (143, 142, 141, 140 toolsets)
- 14.2 2019 BuildTools
- 14.1 2017 BuildTools
- 14.0 2015 BuildTools (via 2022)
Stress tests:
- Configuration:
  - invoked on mapped network drive
  - msvc debug log enabled
- Combinations: 442:
  - Python 3.6.8 x86 embeddable
  - Python 3.6.8 amd64 embeddable
  - Python 3.11.0rc2+ arm
  - Python 3.11.2 arm64 embeddable
runtest.py --all [1271 tests]:
- Configuration:
  - invoked on mapped network drive
  - msvc cache enabled
  - msvc debug log enabled
- WinPython 3.6.8 x86 venv [6 failed, 262 no result]
- WinPython 3.6.8 amd64 venv [6 failed, 262 no result]
- Python 3.11.0rc2+ arm venv [6 failed, 284 no result]
- Python 3.11.2 arm64 venv [6 failed, 262 no result]
- Common failures [6 failed]:
  - Configure [1 failed]:
    - MSVC: Configure test failure on non-AMD64 hosts with default VS 2015+ target architecture #4320
      test\Configure\config-h.py
      < Checking for C function printf()... yes --- > Checking for C function printf()... no
  - virtualenv [5 failed]:
    - test\virtualenv\activated\option\enable-virtualenv.py
    - test\virtualenv\activated\option\ignore-virtualenv.py
    - test\virtualenv\activated\virtualenv_activated_python.py
    - test\virtualenv\activated\virtualenv_detect_virtualenv.py
    - test\virtualenv\unactivated\virtualenv_unactivated_python.py
    - local installation: system (C:), venv (E:), python (E:), source(Z:)
      ValueError: path is on mount 'C:', start on mount 'E:'

Test Env 3 - Windows 10 Pro on AMD64 [VMWare, 4GB]

Visual Studio installations:
- 14.3 2022 Community (143: x86 and amd64 only)
Stress tests:
- Configuration:
  - invoked on mapped network drive
  - msvc debug log enabled
- Combinations: 25:
  - Python 3.11.2 amd64
runtest.py --all [1271 tests]:
- Configuration:
  - invoked on mapped network drive
  - msvc cache enabled
  - msvc debug log enabled
- Python 3.11.2 amd64 [0 failed, 266 no result]

Micellaneous Notes

ARM64

Native ARM64 tools only exist for the 14.3 build tools:

14.3X toolsets: ARM64 tool for ARM64 target:

Input: {}
Where: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\HostARM64\ARM64\cl.exe

14.29 toolsets: AMD64 tool for ARM64 target:

Input: {'MSVC_VERSION': '14.3', 'MSVC_TOOLSET_VERSION': '14.29.30133'}
Where: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\ARM64\cl.exe

Stress Tests

For stress testing, the number of combinations takes into account:

MSVC_VERSION
TARGET_ARCH
MSVC_TOOLSET_VERSION
MSVC_SDK_VERSION
MSVC_UWP_APP
MSVC_SPECTRE_LIBS

…er to be consistent with the current documentation of MSVS_ARCH. Update the documentation for HOST_ARCH.

jcbrill · 2023-03-19T16:48:22Z

Perhaps an MSVC label is warranted.

Last AppVeyor job failure:

Image: Visual Studio 2022; Environment: WINPYTHON=Python311
```
Build execution time has reached the maximum allowed time for your plan (60 minutes).
```
All tests completed and passed prior to timing out.

No additional work for this PR is planned.

mwichmann · 2023-03-19T16:59:16Z

We'll try to get some kind of look at this, thanks for the efforts!

Manually resolve conflicts in CHANGES.txt.

jcbrill · 2023-03-23T14:26:25Z

@mwichmann I need to review the MSVC/MSVS tests. It appears that the host architecture is used in some of the tests.

ARM64 likely needs to be added to tests that use a dictionary to re-map to a canonical host. In addition, the registry probably should be queried due to amd64 builds of python on arm64 have an environment PROCESSOR_ARCHITECTURE of amd64 rather than arm64.

First query the windows registry for the processor architecture and then fallback to the platform machine architecture. The platform.machine() architecture return value for an amd64 build of python (3.6.8) on arm64 is amd64. Update the _ARCH_TO_CANONICAL dictionary from Tool\MSCommon\vc.py.

jcbrill · 2023-03-24T16:21:42Z

Update get_vs_host_arch method for arm64 hosts in testing\framework\TestSConsMSVS.

The boy who cried "Wolf!" says that no additional work is planned ... again.

Manually resolve conflicts in CHANGES.txt.

bdbaddog · 2023-05-04T03:48:30Z

@jcbrill - can you resolve the conflict with SCons/Tool/MSCommon/common.py?

(Sorry for the delay in reviewing. March was consumed by locusts.. ;)

Manually resolve conflicts: * SCons/Tool/MSCommon/common.py

jcbrill · 2023-05-04T12:23:54Z

(Sorry for the delay in reviewing. March was consumed by locusts.. ;)

I hate it when that happens.

Conflicts resolved but there may be "missing" autotyping annotations for the new/modified code.

mwichmann · 2023-05-04T13:01:20Z

Don't worry about missings, I'll sweep through and get at least the "cheap" ones at some point after we get this merged (they were done by a tool).

Manually resolve conflicts in CHANGES.txt and RELEASE.txt.

Manually resolve conflicts in RELEASE.txt.

SCons/Tool/MSCommon/vc.py

bdbaddog · 2023-07-16T19:57:10Z

@jcbrill - finally got back to this. Added a few questions. I think once you get a chance to answer them, should be able to merge fairly quickly.

Thanks for your patience!

jcbrill · 2023-07-17T01:02:33Z

Thanks for your patience!

No worries!

Any and all questions are welcome.

jcbrill added 3 commits March 14, 2023 10:25

Refactor MSCommon/common.py logging filter class due to SConfTests mo…

d352fa5

…dule reset when debug logging is enabled.

Update CHANGES.txt [skip ci]

4eca020

jcbrill changed the title ~~[WIP] Fix 4312 and add Windows on ARM64 support.~~ Fix 4312 and add Windows on ARM64 support. Mar 18, 2023

jcbrill changed the title ~~Fix 4312 and add Windows on ARM64 support.~~ Fix 4312 and add Windows on ARM64 support for msvc. Mar 18, 2023

jcbrill added 2 commits March 19, 2023 09:32

Add arm64 to the MSVS supported architectures list for VS2017 and lat…

26dbadd

…er to be consistent with the current documentation of MSVS_ARCH. Update the documentation for HOST_ARCH.

Update RELEASE.txt. [skip ci]

c50f656

mwichmann added the MSVC Microsoft Visual C++ Support label Mar 19, 2023

jcbrill added 4 commits March 19, 2023 13:39

Fix typographical error in RELEASE.txt [skip ci]

4bdb9de

Fix typographical error in RELEASE.txt [skip ci]

11ec1b4

Fix typographical error in RELEASE.txt [skip ci]

db58ed9

Merge branch 'master' into jbrill-gh4312-fixplus

69fbf56

Manually resolve conflicts in CHANGES.txt.

jcbrill added 2 commits March 31, 2023 09:07

Merge branch 'master' into jbrill-gh4312-fixplus

dad4a5f

Manually resolve conflicts in CHANGES.txt.

Remove placeholders from added text in RELEASE.txt

d3d4229

Merge branch 'master' into jbrill-gh4312-fixplus

20412f3

Manually resolve conflicts: * SCons/Tool/MSCommon/common.py

jcbrill added 2 commits June 20, 2023 08:11

Merge branch 'master' into jbrill-gh4312-fixplus

5e1b9ce

Manually resolve conflicts in CHANGES.txt and RELEASE.txt.

Merge branch 'master' into jbrill-gh4312-fixplus

0f2733f

Manually resolve conflicts in RELEASE.txt.

jcbrill mentioned this pull request Jul 8, 2023

Implement a new subprocess wrapper #4300

Merged

3 tasks

Merge branch 'master' into jbrill-gh4312-fixplus

a2c9937

bdbaddog reviewed Jul 16, 2023

View reviewed changes

SCons/Tool/MSCommon/vc.py Show resolved Hide resolved

bdbaddog reviewed Jul 16, 2023

View reviewed changes

SCons/Tool/MSCommon/vc.py Show resolved Hide resolved

bdbaddog merged commit 461c213 into SCons:master Jul 17, 2023
6 of 7 checks passed

mwichmann added this to the NextRelease milestone Jul 17, 2023

jcbrill deleted the jbrill-gh4312-fixplus branch July 17, 2023 13:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix 4312 and add Windows on ARM64 support for msvc. #4324

Fix 4312 and add Windows on ARM64 support for msvc. #4324

jcbrill commented Mar 14, 2023 •

edited

Loading

jcbrill commented Mar 18, 2023 •

edited

Loading

jcbrill commented Mar 19, 2023

mwichmann commented Mar 19, 2023

jcbrill commented Mar 23, 2023

jcbrill commented Mar 24, 2023

bdbaddog commented May 4, 2023

jcbrill commented May 4, 2023

mwichmann commented May 4, 2023

bdbaddog commented Jul 16, 2023

jcbrill commented Jul 17, 2023

Fix 4312 and add Windows on ARM64 support for msvc. #4324

Fix 4312 and add Windows on ARM64 support for msvc. #4324

Conversation

jcbrill commented Mar 14, 2023 • edited Loading

Contributor Checklist:

jcbrill commented Mar 18, 2023 • edited Loading

Test Environments

Test Env 1 - Windows 10 Pro on AMD64 [AMD Ryzen Threadripper 3960X, 64GB]

Test Env 2 - Windows 11 Pro on ARM64 [Snapdragon 7c, 8GB]

Test Env 3 - Windows 10 Pro on AMD64 [VMWare, 4GB]

Micellaneous Notes

ARM64

Stress Tests

jcbrill commented Mar 19, 2023

mwichmann commented Mar 19, 2023

jcbrill commented Mar 23, 2023

jcbrill commented Mar 24, 2023

bdbaddog commented May 4, 2023

jcbrill commented May 4, 2023

mwichmann commented May 4, 2023

bdbaddog commented Jul 16, 2023

jcbrill commented Jul 17, 2023

jcbrill commented Mar 14, 2023 •

edited

Loading

jcbrill commented Mar 18, 2023 •

edited

Loading