Skip to content

swift test list unexpected sorting behavior #6786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MahdiBM opened this issue Aug 8, 2023 · 4 comments
Closed

swift test list unexpected sorting behavior #6786

MahdiBM opened this issue Aug 8, 2023 · 4 comments
Labels
bug swift test Changes impacting `swift test` tool

Comments

@MahdiBM
Copy link

MahdiBM commented Aug 8, 2023

Description

swift test list's output doesn't keep the sorting that a project's test classes/functions already have.

Reverse Function Order

...
PennyTests.CoinHandlerTests/testUserAtTheEndAndCoinSignAtTheBeginning
PennyTests.GHHooksTests/testEventHandler
PennyTests.GHHooksTests/testHeadingFinder
PennyTests.GHHooksTests/testMakeReleaseBody
PennyTests.GHHooksTests/testMarkdownFormatting
PennyTests.GHHooksTests/testParseCodeOwners
PennyTests.GHHooksTests/testSemVerBump
PennyTests.GHHooksTests/testUnicodesPrefix
PennyTests.GatewayProcessingTests/testAutoFaqsCommand
...

The order of all functions of a class are completely reversed compared to the source files.
This is not the case when Xcode shows the tests list, but it makes the SSWG VSCode extension show the tests in reverse order. (cc @adam-fowler)
For example the correct order for GHHooksTests functions is:

PennyTests.GHHooksTests/testUnicodesPrefix
PennyTests.GHHooksTests/testSemVerBump
PennyTests.GHHooksTests/testParseCodeOwners
PennyTests.GHHooksTests/testMarkdownFormatting
PennyTests.GHHooksTests/testMakeReleaseBody
PennyTests.GHHooksTests/testHeadingFinder
PennyTests.GHHooksTests/testEventHandler

Wrong Class Order

The order of class can also be incorrect compared to the source file orders that both Xcode and VSCode use:

....
PennyTests.CoinHandlerTests/testUserAtTheEndAndCoinSignAtTheBeginning
PennyTests.GHHooksTests/testEventHandler
PennyTests.GHHooksTests/testSemVerBump
PennyTests.GHHooksTests/testUnicodesPrefix
PennyTests.GatewayProcessingTests/testAutoFaqsCommand
PennyTests.GatewayProcessingTests/testAutoPings
PennyTests.GatewayProcessingTests/testBotStateManagerReceivesSignal
PennyTests.GatewayProcessingTests/testCommandsRegisterOnStartup
PennyTests.GatewayProcessingTests/testServerBoostCoins
PennyTests.LeafRenderTests/testAutoPingsHelp
...

As you can see, the PennyTests.GHHooksTests class appears before PennyTests.GatewayProcessingTests.
This is the reverse of what Xcode and VSCode show.
The rest of the class-orders is correct. I've confirmed changing the class's name from GHHo... to GhHo... fixes the order of the list command output, which makes me think the problem is with the priority that the sorting algorithm of the list command gives to uppercase letters.

Expected behavior

Sorting should be consistent with sorting of the source files in IDEs and sorting of the functions that user has chosen in their XCTest classes.

Actual behavior

The sorting mechanism has some unexpected behavior as explained.

Steps to reproduce

You can use the same project to run your tests on: Penny.

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.9.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.87 Apple Swift version 5.9 (swiftlang-5.9.0.124.4 clang-1500.0.38.1)
Target: arm64-apple-macosx14.0
Darwin Macbook.local 23.0.0 Darwin Kernel Version 23.0.0: Tue Jul 18 20:35:35 PDT 2023; root:xnu-10002.0.199.505.1~3/RELEASE_ARM64_T6000 arm64

@MahdiBM MahdiBM added the bug label Aug 8, 2023
@neonichu
Copy link
Contributor

neonichu commented Aug 8, 2023

I don't think I agree with the premise here that swift test should be consistent with Xcode and/or VSCode. If we think a different sort order is better in isolation, I'm open to it, but not because some other tool does it differently.

@MahdiBM
Copy link
Author

MahdiBM commented Aug 8, 2023

So the second Wrong Class Order issue does kind of ask to be inline with how other IDEs work, but that would also be how Finder or probably most other places sort stuff, so it's suboptimal that SPM has chosen to be inconsistent.

Now that i check more, i see the current order of class names seems to be inline with lexicographical ordering, so i guess you could say SPM is at least following something for that.

The first Reverse Function Order issue though is less subjective. The order just doesn't match how users declare their functions.

@adam-fowler
Copy link

@MahdiBM The ordering of tests in VS Code test will be resolved by this swiftlang/sourcekit-lsp#611

@grynspan grynspan added the swift test Changes impacting `swift test` tool label May 7, 2024
@grynspan
Copy link
Contributor

grynspan commented May 7, 2024

swift test does not make any guarantees about the order of reported tests. When using XCTest on macOS (as shipped with Xcode), source location information simply isn't available. SwiftPM theoretically does know the source locations of tests when building with swift-corelibs-xctest, however it is not a goal for that library to implement features that are not available in XCTest-proper.

@grynspan grynspan closed this as not planned Won't fix, can't repro, duplicate, stale May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug swift test Changes impacting `swift test` tool
Projects
None yet
Development

No branches or pull requests

4 participants