Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nightly Test: Test Explorer Suite - Debugs specified XCTest test failed on Test Insiders / Linux (6.0 - jammy) #1317

Open
rbenegal opened this issue Jan 15, 2025 · 3 comments

Comments

@rbenegal
Copy link
Contributor

Test failed with:


Could not find pty 11 on pty host: CodeExpectedError: Could not find pty 11 on pty host
    at M.W (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:45:12233)
    at M.resize (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:45:8773)
    at O.i.<computed> (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:45:2962)
    at Object.call (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:28:4203)
    at Gr.s (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:80677)
    at Gr.q (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:80200)
    at li.value (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:79602)
    at C.B (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:2373)
    at C (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:2443)
    at C.fire (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:2660)
    at MessagePortMain.b (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:24:29136)

...

  1) Test Explorer Suite
       Debugging
         CodeLLDB
           Debugs specified XCTest test:
     Error: Timeout of 300000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/tmp/code/dist/test/integration-tests/testexplorer/TestExplorerIntegration.test.js)
      at listOnTimeout (node:internal/timers:581:17)
      at processTimers (node:internal/timers:519:7)

Link to failed run: https://github.com/swiftlang/vscode-swift/actions/runs/12778956212/job/35622742552

@rbenegal
Copy link
Contributor Author

Another failure in: https://github.com/swiftlang/vscode-swift/actions/runs/12858610127/job/35847958476 (Linux nightly main jammy)

Test logs
  1) Test Explorer Suite
       Debugging
         CodeLLDB
           Debugs specified XCTest test:
     Error: Timeout of 300000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/tmp/code/dist/test/integration-tests/testexplorer/TestExplorerIntegration.test.js)
      at listOnTimeout (node:internal/timers:581:17)
      at processTimers (node:internal/timers:519:7)

  2) Test Explorer Suite
       Standard
         Run Tests (Parallel)
           XCTests (Run Tests (Parallel))
             Runs failing test:

      AssertionError [ERR_ASSERTION]: Expected values to be loosely deep-equal:

{
  errored: [],
  failed: [
    {
      issues: [],
      test: 'PackageTests.FailingXCTestSuite'
    }
  ],
  passed: [
    'PackageTests.FailingXCTestSuite/testFailing'
  ],
  skipped: [],
  unknown: 0
}

should loosely deep-equal

{
  errored: [],
  failed: [
    {
      issues: [
        'failed - oh no'
      ],
      test: 'PackageTests.FailingXCTestSuite/testFailing'
    },
    {
      issues: [],
      test: 'PackageTests.FailingXCTestSuite'
    }
  ],
  passed: [],
  skipped: [],
  unknown: 0
}
      + expected - actual

       {
         "errored": []
         "failed": [
           {
      +      "issues": [
      +        "failed - oh no"
      +      ]
      +      "test": "PackageTests.FailingXCTestSuite/testFailing"
      +    }
      +    {
             "issues": []
             "test": "PackageTests.FailingXCTestSuite"
           }
         ]
      -  "passed": [
      -    "PackageTests.FailingXCTestSuite/testFailing"
      -  ]
      +  "passed": []
         "skipped": []
         "unknown": 0
       }
      
      at assertTestResults (test/integration-tests/testexplorer/utilities.ts:137:12)
      at Context.<anonymous> (test/integration-tests/testexplorer/TestExplorerIntegration.test.ts:746:42)

@rbenegal
Copy link
Contributor Author

Strangely, when I all the integration tests together, the test gets stuck after the swift build stage (when run in Dev Containers for 5.10, since many jobs are now failing consistently on 5.10 as well (for example: https://github.com/swiftlang/vscode-swift/actions/runs/12890321891/job/35942486703?pr=1285)).

  Test Explorer Suite
17:14:34: Activating Swift for Visual Studio Code...
17:14:34: SourceKit-LSP setup
17:14:34: focus: undefined
[7612:0123/171434.691999:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
17:14:34: test/defaultPackage: add: /workspaces/vscode-swift/assets/test/defaultPackage
17:14:34: WARN: Adding package folder file:///workspaces/vscode-swift/assets/test/defaultPackage twice
17:14:35: test/defaultPackage: Listing tests: starting ... 
[7612:0123/171435.194633:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
17:14:35: test/defaultPackage: Listing tests: ... done.
    Debugging
      lldb-dap
        - Debugs specified XCTest test
        - Debugs specified swift-testing test
      CodeLLDB
17:14:35: Exec Task: (test/defaultPackage) swift build --build-tests -Xswiftc -diagnostic-style=llvm -Xswiftc -DTEST_ARGUMENT_SET_VIA_TEST_BUILD_ARGUMENTS_SETTING

However, when I run just that test, it seems to have no problem passing. It does seem like the CodeLLDB extension was update recently (vadimcn/codelldb@v1.11.2...master) so perhaps that could be the cause of flakiness since not much should have changed with Swift 5.10 recently.

Will try to post a PR to move this test to the nightly runs so we can keep an eye on it and not generate false positives for other submissions.

rbenegal added a commit to rbenegal/vscode-swift that referenced this issue Jan 23, 2025
This test case seems to get stuck frequently on swift 5.10 now. Move
it to nightly for now so that we can avoid creating false positives.

Issue: swiftlang#1317
award999 pushed a commit that referenced this issue Jan 23, 2025
This test case seems to get stuck frequently on swift 5.10 now. Move
it to nightly for now so that we can avoid creating false positives.

Issue: #1317
@rbenegal
Copy link
Contributor Author

It seems like setting the lldb.library path (which happens in the test when the full suite runs), causes an issue with the CodeLLDB extension with Swift 5.10, similar to vadimcn/codelldb#1221, where it will fail with:

[ERROR codelldb] Unable to resolve liblldb symbols: "dlsym(0x67663cd0, _ZN4lldb19SBWatchpointOptions21SetWatchpointTypeReadEb): symbol not found"
Error: "\"dlsym(0x67663cd0, _ZN4lldb19SBWatchpointOptions21SetWatchpointTypeReadEb): symbol not found\""
Debug adapter exit code=1 (0x1), signal=null.

or on Linux may fail with something like:

[ERROR codelldb] Unable to resolve liblldb symbols: "/home/user/swift/swift-5.10.1-RELEASE-ubuntu24.04-aarch64/usr/lib/liblldb.so: undefined symbol: _ZN4lldb8SBTarget25WatchpointCreateByAddressEmmNS_19SBWatchpointOptionsERNS_7SBErrorE"
Error: "\"/home/user/swift/swift-5.10.1-RELEASE-ubuntu24.04-aarch64/usr/lib/liblldb.so: undefined symbol: _ZN4lldb8SBTarget25WatchpointCreateByAddressEmmNS_19SBWatchpointOptionsERNS_7SBErrorE\""

Until the extension and/or its dependencies are fixed, it looks like the following temporary workaround options are possible:

  • Install a previous version of CodeLLDB (version 1.11.1 should work, can install by going to Extensions panel > Gear icon > Install Specific Version...)
  • Clear out the lldb.library path in the VSCode settings. Features like getting variable info will not work correctly, and you may receive a prompt to update CodeLLDB settings at startup again the next time you open the project.
  • Move to to a Swift 6.0.3 toolchain by updating Xcode to a newer version or getting a newer toolchain from https://www.swift.org/install/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant