Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test Failure: Test Explorer Suite - XCTests (Run Tests (Parallel)) failed on Test Insiders / Linux (6.0 - jammy) #1334

Open
rbenegal opened this issue Jan 24, 2025 · 0 comments
Open

Test Failure: Test Explorer Suite - XCTests (Run Tests (Parallel)) failed on Test Insiders / Linux (6.0 - jammy) #1334

rbenegal opened this issue Jan 24, 2025 · 0 comments
Labels
testing infrastructure Issues related to CI jobs or extension test cases

Comments

@rbenegal
Copy link
Contributor

Failure logs

1) Test Explorer Suite
       Standard
         Run Tests (Parallel)
           XCTests (Run Tests (Parallel))
             Runs failing test:

      AssertionError [ERR_ASSERTION]: Expected values to be loosely deep-equal:

{
  errored: [],
  failed: [
    {
      issues: [],
      test: 'PackageTests.FailingXCTestSuite'
    }
  ],
  passed: [
    'PackageTests.FailingXCTestSuite/testFailing'
  ],
  skipped: [],
  unknown: 0
}

should loosely deep-equal

{
  errored: [],
  failed: [
    {
      issues: [
        'failed - oh no'
      ],
      test: 'PackageTests.FailingXCTestSuite/testFailing'
    },
    {
      issues: [],
      test: 'PackageTests.FailingXCTestSuite'
    }
  ],
  passed: [],
  skipped: [],
  unknown: 0
}
      + expected - actual

       {
         "errored": []
         "failed": [
           {
      +      "issues": [
      +        "failed - oh no"
      +      ]
      +      "test": "PackageTests.FailingXCTestSuite/testFailing"
      +    }
      +    {
             "issues": []
             "test": "PackageTests.FailingXCTestSuite"
           }
         ]
      -  "passed": [
      -    "PackageTests.FailingXCTestSuite/testFailing"
      -  ]
      +  "passed": []
         "skipped": []
         "unknown": 0
       }
      
      at assertTestResults (test/integration-tests/testexplorer/utilities.ts:137:12)
      at Context.<anonymous> (test/integration-tests/testexplorer/TestExplorerIntegration.test.ts:746:42)

1 test failed.

Link to failed run: https://github.com/swiftlang/vscode-swift/actions/runs/12940729478/job/36095522953
plemarquand added a commit to plemarquand/vscode-swift that referenced this issue Jan 28, 2025
@plemarquand
Output build log on testing related integration test failures
d3e750a 
In service of tracking down flaky test failures like swiftlang#1334, dump the
build/test run log when a testing related integration test fails.
@plemarquand plemarquand mentioned this issue Jan 28, 2025
Merged
plemarquand added a commit to plemarquand/vscode-swift that referenced this issue Jan 28, 2025
@plemarquand
Output build log on testing related integration test failures
be22745 
In service of tracking down flaky test failures like swiftlang#1334, dump the
build/test run log when a testing related integration test fails.
@award999 award999 added the testing infrastructure Issues related to CI jobs or extension test cases label Feb 5, 2025
@award999 award999 moved this from Unscreened to Backlog in Swift Extension for Visual Studio Code Feb 5, 2025
plemarquand added a commit to plemarquand/vscode-swift that referenced this issue Feb 13, 2025
@plemarquand
Fix flakey XCTest parallel test failure reporting
737bb7c 
The `ParallelXCTestRunStateProxy` in `XCTestOutputParser.ts` wraps the
test run state and only allows issues to be recorded using the terminal
output of the test run. The remaining test run state is filled in with
the xml xUnit output. This is because SwiftPM only dumps test output to
the terminal if there is a failure. Using these two sources we can
(almost) fully reconstruct a regular test run.

When parsing XCTest output from the terminal failure messages might be
broken up over multiple lines. In order to capture all these lines and
group them in to a single failure message even if the message is broken
up across terminal buffer chunks we maintain an `excess` variable on the
TestRunState, which we check at the beginning of chunk parsing to know
if we need to continue an error message.

However, the `ParallelXCTestRunStateProxy` that wraps the test run state
was not forwarding along the setting of `excess`. Instead `excess` was
set on the proxy, which is recreated for every chunk of output parsed.

This manifested as XCTests _sometimes_ not correctly reporting their
error message when run in the Parallel test profile, instead reporting
the message "Failed".

Issue: swiftlang#1334
plemarquand added a commit to plemarquand/vscode-swift that referenced this issue Feb 13, 2025
@plemarquand
Fix flakey XCTest parallel test failure reporting
9bab194 
The `ParallelXCTestRunStateProxy` in `XCTestOutputParser.ts` wraps the
test run state and only allows issues to be recorded using the terminal
output of the test run. The remaining test run state is filled in with
the xml xUnit output. This is because SwiftPM only dumps test output to
the terminal if there is a failure. Using these two sources we can
(almost) fully reconstruct a regular test run.

When parsing XCTest output from the terminal failure messages might be
broken up over multiple lines. In order to capture all these lines and
group them in to a single failure message even if the message is broken
up across terminal buffer chunks we maintain an `excess` variable on the
TestRunState, which we check at the beginning of chunk parsing to know
if we need to continue an error message.

However, the `ParallelXCTestRunStateProxy` that wraps the test run state
was not forwarding along the setting of `excess`. Instead `excess` was
set on the proxy, which is recreated for every chunk of output parsed.

This manifested as XCTests _sometimes_ not correctly reporting their
error message when run in the Parallel test profile, instead reporting
the message "Failed".

Issue: swiftlang#1334
plemarquand added a commit that referenced this issue Feb 13, 2025
@plemarquand
Fix flakey XCTest parallel test failure reporting (#1343)
0296cbe 
The `ParallelXCTestRunStateProxy` in `XCTestOutputParser.ts` wraps the
test run state and only allows issues to be recorded using the terminal
output of the test run. The remaining test run state is filled in with
the xml xUnit output. This is because SwiftPM only dumps test output to
the terminal if there is a failure. Using these two sources we can
(almost) fully reconstruct a regular test run.

When parsing XCTest output from the terminal failure messages might be
broken up over multiple lines. In order to capture all these lines and
group them in to a single failure message even if the message is broken
up across terminal buffer chunks we maintain an `excess` variable on the
TestRunState, which we check at the beginning of chunk parsing to know
if we need to continue an error message.

However, the `ParallelXCTestRunStateProxy` that wraps the test run state
was not forwarding along the setting of `excess`. Instead `excess` was
set on the proxy, which is recreated for every chunk of output parsed.

This manifested as XCTests _sometimes_ not correctly reporting their
error message when run in the Parallel test profile, instead reporting
the message "Failed".

Issue: #1334
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
testing infrastructure Issues related to CI jobs or extension test cases
Projects
Swift Extension for Visual Studio Code
Status: Backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@award999 @rbenegal