[Windows] Run Python unit test CI on Windows #13716

GregoryComer · 2025-08-26T23:24:59Z

Run unit test CI on Windows. I've disabled a few jobs (mainly pending tokenizers being available on Windows), but most everything is working out of box.

I did also make a few additional fixes:

Shorten names for a few external project targets in CMake, in order to work around max path length limitations.
Update load code for kernel libraries in python. The binary names do not start with lib by default on Windows. This could be solved in the build, but it seems easier to just extend the glob.
Fix an issue blocking iterative builds for XNNPACK - we need to cause the schema move command to overwrite the existing artifact in the build directory.
Disable tokenizer install on Windows for now. Pending enablement in the tokenizers repo.
Fix paths in setup.py for Windows builds.

[ghstack-poisoned]

GregoryComer · 2025-08-26T23:25:00Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2025-08-26T23:25:03Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13716

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

ROCm MI2xx CI/CD workflows failing due to : download from https://api.github.com/repos/pytorch/pytorch timed out.

❌ 5 New Failures, 4 Unrelated Failures

As of commit fbde97d with merge base 6c1ef96 ():

NEW FAILURES - The following jobs have failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold
pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh)
RuntimeError: Command docker exec -t e103e9866388d5783b2041a4088e01bf6ed9bd983005891d7e1f18f5084f3b37 /exec failed with exit code 1
trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
RuntimeError: Command docker exec -t ebe00fafb9a2ed6c8c9a9d48adc545650fd6c5145f8bd17403d80e950d70bdca /exec failed with exit code 1
trunk / test-arm-backend (test_smaller_stories_llama) / linux-job (gh)
RuntimeError: Command docker exec -t 368a4c5cce1cd067cc802835c1596c0699d5a1f1bd0e6c4f57acc2a2568c6ffc /exec failed with exit code 1
trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 139

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh) (trunk failure)
backends/arm/test/models/stable_diffusion/test_T5EncoderModel.py::TestT5EncoderModel::test_T5EncoderModel_tosa_INT
pull / unittest-nxp-neutron / linux-job (gh) (trunk failure)
test_split_group_convolution__applied_by_default
trunk / test-arm-backend (test_pytest_models_ethosu_fvp) / linux-job (gh) (trunk failure)
backends/arm/test/models/stable_diffusion/test_T5EncoderModel.py::TestT5EncoderModel::test_T5EncoderModel_tosa_INT
trunk / test-huggingface-transformers-coreml (qwen3-1.7b|xnnpack|--quantize) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 139

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: a5ae4bd ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: 82b1993 ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: eeddb89 ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: b29e46d ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: 166e708 ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: 91433dd ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: 1daa38d ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: 3e1d63d ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: b07023b ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

ghstack-source-id: 036019b ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

GregoryComer · 2025-09-02T18:24:18Z

third-party/CMakeLists.txt

 # Similar to flatbuffers, we want to build flatcc for the host. See inline comments
 # in the flatbuffers ExternalProject_Add for more details.
 ExternalProject_Add(
-  flatcc_external_project


This is to resolve path length issues.

mergennachin

See inline

mergennachin · 2025-09-02T19:24:44Z

.ci/scripts/unittest-windows.ps1

+# pytest -n auto --cov=./ --cov-report=xml
+pytest --continue-on-collection-errors -v --full-trace -c pytest-windows.ini -n auto


why --continue-on-collection-errors?

Good catch. This is a holdover from progressive enablement and should not be necessary any more. I've removed it.

mergennachin · 2025-09-02T19:30:36Z

third-party/CMakeLists.txt

-  flatbuffers_external_project
-  PREFIX ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers_external_project
+  flatbuffers_ep
+  PREFIX ${CMAKE_CURRENT_BINARY_DIR}/flatc_proj


isn't it flatbuffers_ep?

This was intentional, though I can rename it if desired. Flatbuffers builds flatc (distinct from flatcc). The path limitations are pretty tight in CI so I squeezed out a few more characters.

mergennachin · 2025-09-02T19:32:30Z

pytest-windows.ini

@@ -0,0 +1,115 @@
+# NOTE: This file is a copy of pytest.ini, but with additional tests disabled for Windows. This
+# is intended to be a short-term solution to allow for incrementally enabling tests on Windows.
+# This file is intended to be deleted once the enablement is complete.


create an issue to track deletion of this file please

Filed as #13883.

[ghstack-poisoned]

GregoryComer · 2025-09-02T21:47:33Z

After rebasing, I do see one test failure on the Windows job that looks like a flake. I'm going to investigate.

To see why this happens see 'Known limitations' in documentation for pytest-xdist
=========================== short test summary info ===========================
ERROR devtools/etrecord/tests/etrecord_test.py - torch._dynamo.exc.InternalTorchDynamoError: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process

mergennachin

Thanks, looks good!

[ghstack-poisoned]

ghstack-source-id: 65f1ac6 ghstack-comment-id: 3226055310 Pull-Request: #13716

[ghstack-poisoned]

GregoryComer · 2025-09-03T04:48:16Z

There appears to be a race condition with the inductor code cache when performing strict mode export from multiple processes on Windows. It's causing occasional test failures on ET, so I'm disabling parallel execution on the Windows unit test jobs for now. I'll see if I can put together a nicer repro and file a bug against PyTorch.

If not having parallel execution is prohibitive, we might be able work around it by not exporting on import in the devtools path, which is what consistently repros the issue during test collection. The race condition will still exist, but the rate at which we hit it might be acceptably low.

[ghstack-poisoned]

jackzhxng

Btw - can you add a release notes: windows and add this to it?

GregoryComer · 2025-09-03T19:06:33Z

I've verified that the remaining CI failures are pre-existing (arm tests, core ML model tests, and NXP tests).

kimishpatel · 2025-09-19T15:18:20Z

kernels/quantized/__init__.py

    from pathlib import Path

-    libs = list(Path(__file__).parent.resolve().glob("**/libquantized_ops_aot_lib.*"))
+    libs = list(Path(__file__).parent.resolve().glob("**/*quantized_ops_aot_lib.*"))


why this change? is lib name something different on windows?

Yeah, it's named quantized_ops_aot_lib.dll (no lib prefix). We could maybe force the dll to have the lib prefix in the filename in cmake, but it seemed more idiomatic to do this.

GregoryComer added 4 commits August 26, 2025 15:48

Update

d6a70d6

[ghstack-poisoned]

Update

18866f4

[ghstack-poisoned]

Update

e7423a4

[ghstack-poisoned]

Update

323d96c

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 26, 2025

Run Python unit test CI on Windows

e007bac

ghstack-source-id: a5ae4bd ghstack-comment-id: 3226055310 Pull-Request: #13716

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 26, 2025

This was referenced Aug 26, 2025

Fix portable library build on Windows #13700

Merged

Fix remaining build issues on Windows, re-enable Windows pybind preset build CI job #13702

Merged

GregoryComer added 3 commits August 26, 2025 19:57

Update

bca23f5

[ghstack-poisoned]

Update

ab69e35

[ghstack-poisoned]

Update

c47d467

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

095d674

ghstack-source-id: 82b1993 ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

cd8e9bc

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

d9d0859

ghstack-source-id: eeddb89 ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

cb60871

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

5eed794

ghstack-source-id: b29e46d ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

19dda19

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

c9f6994

ghstack-source-id: 166e708 ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

cd0178f

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

d1f2468

ghstack-source-id: 91433dd ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

ed4cffd

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

fa2e863

ghstack-source-id: 1daa38d ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

934682f

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

5e5af07

ghstack-source-id: 3e1d63d ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

ef62839

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

90c5ed7

ghstack-source-id: b07023b ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

441fd5f

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Aug 27, 2025

Run Python unit test CI on Windows

5657676

ghstack-source-id: 036019b ghstack-comment-id: 3226055310 Pull-Request: #13716

Update

2706db0

[ghstack-poisoned]

GregoryComer commented Sep 2, 2025

View reviewed changes

GregoryComer marked this pull request as ready for review September 2, 2025 18:24

GregoryComer requested review from larryliu0820, kirklandsign, manuelcandales, swolchok, jackzhxng, lucylq, shoumikhin and cccclai as code owners September 2, 2025 18:24

mergennachin reviewed Sep 2, 2025

View reviewed changes

Update

fdd8e12

[ghstack-poisoned]

GregoryComer mentioned this pull request Sep 2, 2025

Enable all Windows tests / remove pytest-windows.ini #13883

Open

15 tasks

mergennachin approved these changes Sep 2, 2025

View reviewed changes

GregoryComer added 2 commits September 2, 2025 17:59

Update

edfd321

[ghstack-poisoned]

Update

b41ddee

[ghstack-poisoned]

GregoryComer added a commit that referenced this pull request Sep 3, 2025

Run Python unit test CI on Windows

5af8e1e

ghstack-source-id: 65f1ac6 ghstack-comment-id: 3226055310 Pull-Request: #13716

GregoryComer added 2 commits September 2, 2025 21:06

Update

fd3d9c3

[ghstack-poisoned]

Update

8a4a0e3

[ghstack-poisoned]

GregoryComer requested a review from metascroy as a code owner September 3, 2025 04:41

Update

fbde97d

[ghstack-poisoned]

jackzhxng reviewed Sep 3, 2025

View reviewed changes

GregoryComer added the release notes: windows label Sep 3, 2025

GregoryComer merged commit d7fd78b into main Sep 3, 2025
243 of 252 checks passed

GregoryComer deleted the gh/GregoryComer/146/head branch September 3, 2025 19:06

kimishpatel reviewed Sep 19, 2025

View reviewed changes

		# pytest -n auto --cov=./ --cov-report=xml
		pytest --continue-on-collection-errors -v --full-trace -c pytest-windows.ini -n auto

[Windows] Run Python unit test CI on Windows #13716

[Windows] Run Python unit test CI on Windows #13716

Uh oh!

Conversation

GregoryComer commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GregoryComer commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13716

❗ 1 Active SEVs

❌ 5 New Failures, 4 Unrelated Failures

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergennachin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GregoryComer commented Sep 2, 2025

Uh oh!

mergennachin left a comment

Choose a reason for hiding this comment

Uh oh!

GregoryComer commented Sep 3, 2025

Uh oh!

jackzhxng left a comment

Choose a reason for hiding this comment

Uh oh!

GregoryComer commented Sep 3, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GregoryComer commented Aug 26, 2025 •

edited

Loading

GregoryComer commented Aug 26, 2025 •

edited

Loading

pytorch-bot bot commented Aug 26, 2025 •

edited

Loading