[build.ps1] Enable LLDB tests in Windows builds #76894

weliveindetail · 2024-10-07T19:08:37Z

This PR adds handling for the -Test lldb option, which allows to enable the LLDB test suite on Windows builds of the swift toolchain, fixing swiftlang/llvm-project#9141

While we still see a lot of test failures, it's time to give the CI a try. Once we get to a stable subset of tests that pass, we can iterate to enable the remaining ones and increase coverage.

The PR adds a local Windows-specific mechanism to mark bulks of tests as XFAIL or SKIPPED. These tests are listed in file utils/windows-llvm-lit-test-overrides.txt. The goal is to get it down to a low number and remove the mechanism and file eventually.

weliveindetail · 2024-10-07T19:09:41Z

This is a preview. We need to land swiftlang/llvm-project#9370 first.

utils/build.ps1

weliveindetail · 2024-10-16T16:07:25Z

@compnerd This should be ready for testing with the companion PR linked above. Do we need to land that or can we ask the CI to checkout llvm-project from that PR instead of the original stable/20230725?

compnerd · 2024-10-16T23:30:23Z

Please test with following PRs:
swiftlang/llvm-project#9370

@swift-ci please test windows platform

weliveindetail · 2024-10-17T10:23:20Z

Oh wow, that works indeed! And sorry, it was easy to confuse. We need the latest linked PR. The other one was merged in the meantime :) But not too bad, I anyway had to patch build-windows-toolchain.bat.

@compnerd I checked locally and I think that the following will work:

Please test with following PRs:
https://github.com/swiftlang/llvm-project/pull/9429

@swift-ci please test windows platform

hjyamauchi · 2024-10-23T15:45:38Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test windows platform

weliveindetail · 2024-10-24T08:23:46Z

Great, thanks. Good news is that build.ps1 now runs with testing enabled for LLDB. Learning is that the build machine doesn't have winget, so we have to find another way to install make.exe from [GnuWin32]:(https://gnuwin32.sourceforge.net/packages/make.htm):

swift\utils\build.ps1 : The term 'winget' is not recognized as 
the name of a cmdlet, function, script file, or operable program.

Edit: We can just download and install it like we do for Python, WiX, FlexBison, etc.

hjyamauchi · 2024-10-24T16:42:30Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test windows platform

weliveindetail · 2024-10-24T19:13:59Z

The build failed with the following error in build step cmake.exe --build T:\5 --target distribution:

[4797/6491][ 73%][1334.403s] Linking Swift static library lib\swiftASTGen.lib
...
swift\lib\ASTGen\Sources\ASTGen\SourceFile.swift:69:45: error: type 'Parser.ExperimentalFeatures' has no member 'sendingArgsAndResults'
    mapFeature(.SendingArgsAndResults, to: .sendingArgsAndResults)
                                           ~^~~~~~~~~~~~~~~~~~~~~

This doesn't appear to be related to my changes. Is it due to the merge conflict in utils/build-windows-toolchain.bat maybe? It's not related either, but does it cause the bot to silently skip the merge and build my (outdated) branch against newer dependencies? Let me resolve that.

hjyamauchi · 2024-10-24T19:17:44Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

weliveindetail · 2024-10-25T07:15:15Z

Great, tests are running!

Testing Time: 202.24s
  Excluded           :   18
  Unsupported        : 1187
  Passed             : 2058
  Expectedly Failed  :   45
  Unresolved         :    1
  Failed             :   56
  Unexpectedly Passed:    4

SwiftREPL failures (28) will be fixed with swiftlang/llvm-project#9416, the others I will investigate today.

weliveindetail · 2024-10-25T10:49:45Z

In these NativePDB tests, a line breakpoint is set and it gets resolved to a location that seems right, but at startup time we add another location and that fails the test. This might be an actual bug:

  lldb-shell :: SymbolFile/NativePDB/local-variables.cpp
  lldb-shell :: SymbolFile/NativePDB/stack_unwinding01.cpp

The tests in 25f5eca should be fixed and they pass reliably on my machine, but they failed in the CI test run. Let's see if they XPASS next time, because the companion PR in llvm-project now has the SDKROOT patch: swiftlang/llvm-project@62b9626 Otherwise we might be missing a cherry-pick.

And for the record: The following test failed but the detailed report is lacking in the log! 🤷‍♂️ This is the only unreasonable difference to the log from the companion PR.

  lldb-api :: source-manager/TestSourceManager.py

weliveindetail · 2024-10-25T10:58:22Z

The latest two commits integrate the changes from PRs #76974 and #77005 into this one. I would usually prefer small isolated changes, but the per-merge infrastructure makes it unbearable. I will abaondon the other PRs and only continue with this one.

hjyamauchi · 2024-10-25T16:30:28Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

weliveindetail · 2024-10-28T10:06:18Z

SwiftREPL tests keep failing:

$ "t:\5\bin\lldb.exe" "--no-lldbinit" "-S" "T:/5/tools/lldb\test\Shell\lit-lldb-init-quiet" "--repl"
error: failed to stop process at REPL breakpoint

This means that the dummy executable for the REPL didn't load. It usually happens if it lacks a runtime dependency like swiftCore: https://github.com/swiftlang/llvm-project/blob/stable/20230725/lldb/tools/repl/swift/main.c#L66

weliveindetail · 2024-10-28T10:08:42Z

Otherwise only windows-unaligned-x86_64 failed this time:

llvm-project\lldb\test\Shell\Unwind\windows-unaligned-x86_64.test:20:10: error: CHECK: expected string not found in input
# CHECK: stop reason = breakpoint 1.1
         ^
<stdin>:9:61: note: scanning from here
Breakpoint 1: where = windows-unaligned-x86_64.test.tmp`func at windows-unaligned-x86_64.cpp:3, address = 0x0000000140001000
                                                            ^
<stdin>:13:14: note: possible intended match here
* thread #1, stop reason = breakpoint 1.2
             ^

This looks very similar to the problem we saw in the NativePDB tests last time. windows-unaligned-x86_64 would have failed for the same reason already, but it showed up as UNSUPPORTED back then. This indicates a flake at build-time of the inferior. It would be a pity if we had to skip it due to these issues, because there is a good chance that the actual bug behind it was fixed with llvm/llvm-project#101710

hjyamauchi · 2024-10-28T15:11:06Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

hjyamauchi · 2024-11-04T17:19:16Z

Please test with following PRs:
swiftlang/llvm-project#9429

@swift-ci please test

weliveindetail · 2024-11-05T08:21:28Z

The builds failed with compile errors that indicate a LLVM version mismatch. I guess we have to build against stable/20240723 now. I will prepare a new companion PR for that.

weliveindetail · 2024-11-05T16:45:26Z

Can we run again with this PR please? This time we should reach the tests.
swiftlang/llvm-project#9513

hjyamauchi · 2024-11-05T17:13:33Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test

utils/build.ps1

weliveindetail · 2024-11-06T10:04:55Z

@compnerd Thanks for your feedback.
@hjyamauchi Unfortunately, the bots didn't run. Can we try again please? We only need Windows platform right now.

Please test with following PRs:
https://github.com/swiftlang/llvm-project/pull/9513

@swift-ci please test windows platform

weliveindetail · 2024-11-21T22:59:41Z

No unexpected failures or passes. This should have been the first successful run:

Testing Time: 222.85s

Total Discovered Tests: 3421
  Excluded         :   20 (0.58%)
  Unsupported      : 1227 (35.87%)
  Passed           : 2055 (60.07%)
  Expectedly Failed:  116 (3.39%)
  Unresolved       :    3 (0.09%)

But still Error: cmake.exe exited with code 1. To be investigated next week.

hjyamauchi · 2024-11-26T19:21:23Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test windows platform

weliveindetail · 2024-11-27T10:04:29Z

Finally, we have a successful run!

Total Discovered Tests: 3422
  Excluded         :   23 (0.67%)
  Unsupported      : 1228 (35.89%)
  Passed           : 2055 (60.05%)
  Expectedly Failed:  116 (3.39%)

Runtime impact on the CI pipeline was 5min: 3m10s for running the actual tests and ~2min for building LLDB test dependencies. Timing of the testing stage compared to the two surrounding successful builds:

    #33729: 19:25 - 19:57 = 32min (cxx interop)
--> #33732: 21:57 - 22:34 = 37min (enable lldb tests)
    #33733: 23:41 - 24:13 = 32min (enable libxml2)

hjyamauchi · 2024-11-27T18:01:04Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test

weliveindetail · 2024-11-27T20:52:22Z

Build failed for an unrelated reason. Didn't reach the test stage.

hjyamauchi · 2024-11-27T22:10:40Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test windows platform

compnerd · 2024-12-02T20:30:22Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test Windows platform

weliveindetail · 2024-12-03T11:07:03Z

Tests finished successfully. Again no falkes, same results as last run, same timing:

#33837 23:00 - 23:37 = 37min

hjyamauchi · 2024-12-03T17:43:40Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test Windows platform

hjyamauchi · 2024-12-03T17:44:23Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test

weliveindetail · 2024-12-06T12:25:41Z

Rebase and fix conflicts after #77948

compnerd · 2024-12-06T16:28:01Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test

weliveindetail · 2024-12-06T21:57:49Z

The Windows build failed due to one test in LLDB:

======================================================================
FAIL: test_compressed_section_data (TestSectionAPI.SectionAPITestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\llvm-project\lldb\packages\Python\lldbsuite\test\decorators.py", line 452, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\llvm-project\lldb\test\API\python_api\section\TestSectionAPI.py", line 67, in test_compressed_section_data
    self.assertEqual(section_data, [0x20, 0x30, 0x40, 0x50, 0x60, 0x70, 0x80, 0x90])
AssertionError: Lists differ: [] != [32, 48, 64, 80, 96, 112, 128, 144]

Apparently, we got an empty list from section.GetSectionData(): https://github.com/swiftlang/llvm-project/blob/22b89865683de106191106cc504cda80db6d4dd1/lldb/test/API/python_api/section/TestSectionAPI.py#L66-L67

I never saw this test failing and I don't see a recent change, that might have caused it. Needs investigation. I will file an issue and disable it for the time being.

compnerd · 2024-12-06T22:11:03Z

Please test with following PRs:
swiftlang/llvm-project#9513

@swift-ci please test

weliveindetail · 2024-12-07T21:05:57Z

LLDB tests finished successfully. No falkes. The build failed in the subsequent check-swift step, where the LLVM build-tree is reconfigured and LibXml2 can't be found.

weliveindetail · 2025-01-06T13:47:39Z

@swift-ci please test

weliveindetail · 2025-01-07T10:10:44Z

The Windows bot didn't run, let's try again.

@swift-ci please test Windows platform

weliveindetail · 2025-01-07T13:44:10Z

Two flakes: first new, second known (we'd have to adjust the filter here, because it doesn't work for individual unittests)

Failed Tests (2):
  lldb-api :: commands/statistics/basic/TestStats.py
  lldb-unit :: Host/PipeTest/WriteWithTimeout

weliveindetail · 2025-01-07T13:49:02Z

@swift-ci please test Windows platform

weliveindetail · 2025-01-08T08:47:08Z

Tests passed. Let's see if this is reproducible.

@swift-ci please test Windows platform

weliveindetail · 2025-01-08T11:10:42Z

The build failed with an unrelated error in LLDB:

lldb\source\Plugins\LanguageRuntime\Swift\SwiftLanguageRuntimeDynamicTypeResolution.cpp(324): error C2039: 'MangleClass': is not a member of 'lldb_private::swift_demangle'

weliveindetail · 2025-01-08T16:20:56Z

#78488 should be fixed. Let's try again.

@swift-ci please test Windows platform

weliveindetail · 2025-01-09T08:52:03Z

Tests passed. Let's see if this is reproducible.

@swift-ci please test Windows platform

weliveindetail commented Oct 10, 2024

View reviewed changes

utils/build.ps1 Outdated Show resolved Hide resolved

weliveindetail commented Oct 16, 2024

View reviewed changes

utils/build.ps1 Outdated Show resolved Hide resolved

weliveindetail mentioned this pull request Oct 16, 2024

🍒 Cherrypicks to allow Swift to enable LLDB tests in Windows builds swiftlang/llvm-project#9429

Closed

weliveindetail marked this pull request as ready for review October 16, 2024 16:04

weliveindetail requested a review from compnerd as a code owner October 16, 2024 16:04

weliveindetail requested review from artemcm and tshortli as code owners November 4, 2024 14:54

weliveindetail mentioned this pull request Nov 5, 2024

Allow Swift to enable LLDB tests in Windows CI swiftlang/llvm-project#9513

Merged

compnerd reviewed Nov 5, 2024

View reviewed changes

utils/build.ps1 Outdated Show resolved Hide resolved

utils/build.ps1 Outdated Show resolved Hide resolved

utils/build.ps1 Outdated Show resolved Hide resolved

utils/build.ps1 Outdated Show resolved Hide resolved

weliveindetail force-pushed the kendal/lldb-tests branch from 8e7e50d to e99b3ec Compare November 27, 2024 10:44

weliveindetail force-pushed the kendal/lldb-tests branch from e99b3ec to 44099ab Compare December 3, 2024 11:19

weliveindetail force-pushed the kendal/lldb-tests branch from 44099ab to ef2c0af Compare December 6, 2024 12:25

weliveindetail mentioned this pull request Dec 6, 2024

LLDB test TestSectionAPI sporadicly fails in CI on Windows swiftlang/llvm-project#9705

Open

[windows] Run LLDB Tests as part of the toolchain build

0349f3e

weliveindetail force-pushed the kendal/lldb-tests branch from cd81c57 to 0349f3e Compare January 6, 2025 13:47

Add failed tests to filter

a65e842

[build.ps1] Enable LLDB tests in Windows builds #76894

Are you sure you want to change the base?

[build.ps1] Enable LLDB tests in Windows builds #76894

Conversation

weliveindetail commented Oct 7, 2024

weliveindetail commented Oct 7, 2024

weliveindetail commented Oct 16, 2024

compnerd commented Oct 16, 2024

weliveindetail commented Oct 17, 2024

hjyamauchi commented Oct 23, 2024

weliveindetail commented Oct 24, 2024 • edited Loading

hjyamauchi commented Oct 24, 2024

weliveindetail commented Oct 24, 2024

hjyamauchi commented Oct 24, 2024

weliveindetail commented Oct 25, 2024 • edited Loading

weliveindetail commented Oct 25, 2024

weliveindetail commented Oct 25, 2024

hjyamauchi commented Oct 25, 2024

weliveindetail commented Oct 28, 2024

weliveindetail commented Oct 28, 2024

hjyamauchi commented Oct 28, 2024

hjyamauchi commented Nov 4, 2024

weliveindetail commented Nov 5, 2024

weliveindetail commented Nov 5, 2024

hjyamauchi commented Nov 5, 2024

weliveindetail commented Nov 6, 2024

weliveindetail commented Nov 21, 2024

hjyamauchi commented Nov 26, 2024

weliveindetail commented Nov 27, 2024

hjyamauchi commented Nov 27, 2024

weliveindetail commented Nov 27, 2024

hjyamauchi commented Nov 27, 2024

compnerd commented Dec 2, 2024

weliveindetail commented Dec 3, 2024

hjyamauchi commented Dec 3, 2024

hjyamauchi commented Dec 3, 2024

weliveindetail commented Dec 6, 2024

compnerd commented Dec 6, 2024

weliveindetail commented Dec 6, 2024

compnerd commented Dec 6, 2024

weliveindetail commented Dec 7, 2024

weliveindetail commented Jan 6, 2025

weliveindetail commented Jan 7, 2025

weliveindetail commented Jan 7, 2025

weliveindetail commented Jan 7, 2025

weliveindetail commented Jan 8, 2025

weliveindetail commented Jan 8, 2025

weliveindetail commented Jan 8, 2025

weliveindetail commented Jan 9, 2025

weliveindetail commented Oct 24, 2024 •

edited

Loading

weliveindetail commented Oct 25, 2024 •

edited

Loading