Testing macros in the OSS toolchain snapshots for macOS don't seem to work with SDK bundles

[finagolfin]

Description

I recently added the Testing library to my Android SDK bundles for Swift 6.1 and 6.2, finagolfin/swift-android-sdk@eb1567b1c, and it works well to cross-compile Testing tests using the latest linux snapshot toolchain and my CI-generated 6.1 SDK bundle, which I checked by cross-compiling @tayloraswift's swift-png and its tests. 😄

However, @marcprux tells me the same didn't work with the OSS mac toolchain snapshot when he tried it with another Testing repo, as he got errors saying:

error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found

suggesting there is some issue with how these Testing macros are installed on macOS.

Reproduction

Follow the instructions to use my Android SDK bundle on macOS, except using the 6.1 bundle linked above, not 6.0.3 as I have not backported the Android patches in this repo to that tag. An alternative would be to try the linux SDK bundle generated by swift-sdk-generator, if it supports this Testing library.

Expected behavior

Testing tests cross-compile fine using the mac toolchain's prebuilt macros, as they do on linux.

Environment

MacOS

Additional information

I have not reproduced this myself. Marc said he'd file this himself, but I'm doing it now so we don't forget and so somebody behind this repo can take a look in the meantime.

[grynspan]

jon@JGStudio swift-PR-76034-1812.xctoolchain % find . -name 'libTestingMacros.dylib'
./usr/lib/swift/host/plugins/testing/libTestingMacros.dylib

So the library is being built, but our CMake script installs it to a different location than it does on Linux. I'm not sure why that is or if it might be the root cause. @stmontgomery @rintaro does either of you have any insight?

[finagolfin]

Oh, Marc also told me there's a ~/Library/Developer/Toolchains/swift-6.0.3-RELEASE.xctoolchain/usr/lib/swift/host/plugins/testing/libTestingMacros.dylib in the OSS toolchain for macOS, but maybe it is deficient or misconfigured in some way.

[grynspan]

Also I don't see a .swiftmodule for TestingMacros. Dunno if that is needed in this context.

[finagolfin]

Also I don't see a .swiftmodule for TestingMacros. Dunno if that is needed in this context.

I don't think it is, as the upcoming 6.1 toolchain for linux only has swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a-fedora39/usr/lib/swift/linux/Testing.swiftmodule/x86_64-unknown-linux-gnu.swiftmodule and the aforementioned plugin usr/lib/swift/host/plugins/libTestingMacros.so, works fine with my 6.1 Android SDK bundle.

[stmontgomery]

Looks like this is, or at least was, a bug in Swift Package Manager: in 6.1, it was checking whether the target triple represented macOS, and only then included the necessary macro plugin search paths to locate libTestingMacros.dylib in macOS toolchains. I think that was happening here (link from the release/6.1 branch):

https://github.com/swiftlang/swift-package-manager/blob/release/6.1/Sources/PackageModel/UserToolchain.swift#L676-L684

However, this code all changed somewhat recently in #8295 and it might have affected this, or perhaps even fixed the issue. I'm not certain, but it would be worth trying using a main development snapshot toolchain which has that change to see.

Regardless, I'm going to transfer this to SwiftPM since it's something controlled at that layer. It may be a dupe, though.

[finagolfin]

Thanks for likely finding the culprit, but my read of the code is that it is still broken in trunk for cross-compilation to non-mac platforms, because it checks the target triple instead of the host triple before applying the macOS plugin paths:

        self.targetTriple = triple

        var swiftCompilerFlags: [String] = []
        var extraLinkerFlags: [String] = []

        let swiftTestingPath: AbsolutePath? = try Self.deriveSwiftTestingPath(
            derivedSwiftCompiler: swiftCompilers.compile,
            swiftSDK: self.swiftSDK,
            triple: triple,
            environment: environment,
            fileSystem: fileSystem
        )

        if triple.isMacOSX, let swiftTestingPath {
            // swift-testing in CommandLineTools, needs extra frameworks search path
            if swiftTestingPath.extension == "framework" {
                swiftCompilerFlags += ["-F", swiftTestingPath.pathString]
            }

            // Otherwise we must have a custom toolchain, add overrides to find its swift-testing ahead of any in the
            // SDK. We expect the library to be in `lib/swift/macosx/testing` and the plugin in
            // `lib/swift/host/plugins/testing`
            if let pluginsPath = try? AbsolutePath(
                validating: "../../host/plugins/testing",
                relativeTo: swiftTestingPath
            ) {
                swiftCompilerFlags += [
                    "-I", swiftTestingPath.pathString,
                    "-L", swiftTestingPath.pathString,
                    "-plugin-path", pluginsPath.pathString,
                ]
            }
        }

I think this was broken in #7920 last fall by @xedin, ironically when he was trying to fix cross-compilation to wasm. The issue appears to be that deriveSwiftTestingPath is used to determine both host and target paths, so it will need to be split up to determine where these host plugins are separately from the target compilation flags, ie checking both host and target triples separately.

[marcprux]

Just to add some data, I have a bare-bones repository https://github.com/marcprux/swift-testing-cross-compile-demo with CI that uses the same Android SDK on both macOS and Ubuntu. You can see the test build output for both hosts at: https://github.com/marcprux/swift-testing-cross-compile-demo/actions/runs/13816030659. Ubuntu passes, but macOS fails:

Building for debugging...
[0/13] Write sources
[2/13] Write swift-version--451CF4548FE66429.txt
[4/15] Compiling TestingCrossCompile TestingCrossCompile.swift
[5/15] Emitting module TestingCrossCompile
[6/16] Wrapping AST for TestingCrossCompile for debugging
error: emit-module command failed with exit code 1 (use -v to see invocation)
[8/18] Emitting module TestingCrossCompileTests
/Users/runner/work/swift-testing-cross-compile-demo/swift-testing-cross-compile-demo/Tests/TestingCrossCompileTests/TestingCrossCompileTests.swift:4:12: error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
2 | @testable import TestingCrossCompile
3 | 
4 | @Test func example() async throws {
  |            `- error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
5 |     // Write your test here and use APIs like `#expect(...)` to check expected conditions.
6 | }

Testing.Test:1:30: note: 'Test' declared here
1 | @attached(peer) public macro Test(_ traits: any TestTrait...) = #externalMacro(module: "TestingMacros", type: "TestDeclarationMacro")
  |                              `- note: 'Test' declared here
[9/18] Compiling TestingCrossCompileTests TestingCrossCompileTests.swift
/Users/runner/work/swift-testing-cross-compile-demo/swift-testing-cross-compile-demo/Tests/TestingCrossCompileTests/TestingCrossCompileTests.swift:4:12: error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
2 | @testable import TestingCrossCompile
3 | 
4 | @Test func example() async throws {
  |            `- error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
5 |     // Write your test here and use APIs like `#expect(...)` to check expected conditions.
6 | }

Testing.Test:1:30: note: 'Test' declared here
1 | @attached(peer) public macro Test(_ traits: any TestTrait...) = #externalMacro(module: "TestingMacros", type: "TestDeclarationMacro")
  |                              `- note: 'Test' declared here

The locally-installed testing libraries on macOS are:

zap Toolchains/swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a.xctoolchain % find . -name '*Test*dylib'
./usr/lib/swift/host/plugins/testing/libTestingMacros.dylib
./usr/lib/swift/macosx/testing/libTesting.dylib

And on Ubuntu are:

marcprux@skipbox:~/.swiftpm/toolchains/swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a-ubuntu24.04$ find . -name '*Test*.so'
./usr/lib/swift/linux/libTesting.so
./usr/lib/swift/linux/libXCTest.so
./usr/lib/swift/host/plugins/libTestingMacros.so

Let me know if there is any jiggery-pokery I can do with file paths or toolchain config to experiment with getting it working.

[stmontgomery]

Just as a temporary experiment, you could try moving or copying ./usr/lib/swift/host/plugins/testing/libTestingMacros.dylib to ./usr/lib/swift/host/plugins (up one level from the testing/ directory) to confirm my theory that it's a search path issue.

[finagolfin]

My take is that it is incorrectly not passing in the -plugin-path when cross-compiling from macOS to other platforms. So first build it natively for macOS with -v, find the -plugin-path flag that ends with /testing, then pass it in to the cross-compile manually with -Xswiftc -plugin-path -Xswiftc <found-path>/testing. That should get it to find the Testing macros at least when cross-compiling, though it may still fail later if missing other flags too.

[marcprux]

Just as a temporary experiment, you could try moving or copying ./usr/lib/swift/host/plugins/testing/libTestingMacros.dylib to ./usr/lib/swift/host/plugins (up one level from the testing/ directory) to confirm my theory that it's a search path issue.

That worked!

zap Toolchains/swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a.xctoolchain % cp usr/lib/swift/host/plugins/testing/libTestingMacros.dylib usr/lib/swift/host/plugins/

zap marcprux/swift-testing-cross-compile-demo % ~/Library/Developer/Toolchains/swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a.xctoolchain/usr/bin/swift build --swift-sdk x86_64-unknown-linux-android24 --build-tests
Building for debugging...
[20/20] Linking TestingCrossCompilePackageTests.xctest
Build complete! (4.70s)

zap marcprux/swift-testing-cross-compile-demo % file ./.build/x86_64-unknown-linux-android24/debug/TestingCrossCompilePackageTests.xctest

./.build/x86_64-unknown-linux-android24/debug/TestingCrossCompilePackageTests.xctest: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, BuildID[sha1]=98e1897f2e50e8ac19473d0a8fd41f2e6349ee30, with debug_info, not stripped

I've also updated swift-android-action to apply this workaround, so now the testing demo is working on both macOS and Ubuntu: https://github.com/marcprux/swift-testing-cross-compile-demo/actions/runs/13819400842

I will "un-apply" the workaround once the fix is available in the macOS toolchain.

[marcprux]

While the test package is now cross-compiling for Android with the aforementioned workaround, they are not actually getting executed by the TestingCrossCompilePackageTests.xctest command in Android. E.g., see https://github.com/marcprux/swift-testing-cross-compile-demo/actions/runs/13820440573, which shows an (intentional) failure when running locally, but passes when running against Android, regardless of whether it was built on macOS or Ubuntu.

This is likely a separate issue, but I'm mentioning it here to point out that fixing this issue won't entirely get testing working. @finagolfin, when you ran the swift-png tests, do you have evidence that the tests were actually executed? E.g., if you add in an intentional failure, do they fail as you would expect?

[stmontgomery]

I think a better short-term workaround may be manually passing the plugin search path using -Xswiftc -plugin-path -Xswiftc <found-path>/testing as @finagolfin suggested, rather than moving content in the toolchain.

[grynspan]

@marcprux I'm not that familiar with GitHub CI, but it looks like it's outright not running the .xctest executable a second time (which is how Swift Testing tests run.)

[grynspan]

Ah, yes:

+ adb shell 'cd /data/local/tmp/android-xctest &&  ./TestingCrossCompilePackageTests.xctest '

Your script needs to also run ./TestingCrossCompilePackageTests.xctest --testing-library swift-testing (or however you might spell that in the script.) Note this command is different from what you'd do when running tests on macOS.

[marcprux]

I think a better short-term workaround may be manually passing the plugin search path using -Xswiftc -plugin-path -Xswiftc <found-path>/testing as @finagolfin suggested, rather than moving content in the toolchain.

That works too. I'll use that instead. Thanks.

[marcprux]

Your script needs to also run ./TestingCrossCompilePackageTests.xctest --testing-library swift-testing (or however you might spell that in the script.) Note this command is different from what you'd do when running tests on macOS.

Thanks, that works as expected (in that the expected failure is now occurring).

@grynspan, do I understand correctly that running ./TestingCrossCompilePackageTests.xctest --testing-library swift-testing will run only the Testing tests, and running without the --testing-library swift-testing flag will run only the non-Testing (i.e., XCTest) tests?

If so, I now need to figure out when I should run the tests with that argument, since I don't want to invoke the tests a second time when all they contain are XCTest test cases. I'm guessing I'll just run patchelf --print-needed TestingCrossCompilePackageTests.xctest and see if the dependencies list contains libTesting.so, unless anyone has a cleverer idea…

[grynspan]

@grynspan, do I understand correctly that running ./TestingCrossCompilePackageTests.xctest --testing-library swift-testing will run only the Testing tests, and running without the --testing-library swift-testing flag will run only the non-Testing (i.e., XCTest) tests?

Correct. The tool basically has two separate main() functions, one per library, because (at this time) they don't know how to talk to each other and can't coordinate their main() function invocations.

If so, I now need to figure out when I should run the tests with that argument, since I don't want to invoke the tests a second time when all they contain are XCTest test cases. I'm guessing I'll just run patchelf --print-needed TestingCrossCompilePackageTests.xctest and see if the dependencies list contains libTesting.so, unless anyone has a cleverer idea…

If you run the two commands, they won't (shouldn't!) overlap output. :)

[grynspan]

Note also that if there are no Swift Testing tests, the library terminates the test process with a specific error code (EX_UNAVAILABLE) to indicate this to the caller, and you probably don't want to treat that code as a failure. XCTest, on the other hand, just does nothing and returns EXIT_SUCCESS.

This is all, of course, normally handled by SwiftPM. The exact details of how SwiftPM and Swift Testing interoperate will be subject to change over time as we build out more of our infrastructure.

[marcprux]

Note also that if there are no Swift Testing tests, the library terminates the test process with a specific error code (EX_UNAVAILABLE) to indicate this to the caller, and you probably don't want to treat that code as a failure.

I'm assuming that this is what I'm seeing when I run a test suite that doesn't use Testing (e.g., Yams), which outputs this error:

Error: Invalid option "--testing-library"
Usage: YamsPackageTests.xctest [OPTION]
       YamsPackageTests.xctest [TESTCASE]

So I think I will want to probe the binary to see if it links to Testing in order to avoid that, since I expect it will be easier than special-casing certain exit codes to count as passing.

[grynspan]

I'm assuming that this is what I'm seeing when I run a test suite that doesn't use Testing (e.g., Yams), which outputs this error:

Error: Invalid option "--testing-library"
Usage: YamsPackageTests.xctest [OPTION]
YamsPackageTests.xctest [TESTCASE]

This is a different problem. For historical reasons, test executables on Linux/etc. can specify a custom main function which prevents SwiftPM from emitting its own main function, and Yams is doing that here. I'd love to remove that feature but there are still a lot of packages that rely on it.

You could still run into this problem if the binary does link to Testing.

[grynspan]

(By the end of this, you'll have reimplemented a third of SwiftPM. 😬)

[grynspan]

Spitballing here, perhaps we could teach swift test to create a shell script (batch file on Windows?) that does approximately what it normally does. SwiftPM has enough information to know if it needs to invoke the executable a second time.

[marcprux]

You could still run into this problem if the binary does link to Testing.

Yeah, I just ran into that when testing swift-algorithms.

My current solution (skiptools/swift-android-action@9a5ef58) seems to be working, although it ain't pretty:

adb shell 'cd /data/local/tmp/android-xctest &&  ./swift-algorithmsPackageTests.xctest  &&  ./swift-algorithmsPackageTests.xctest  --testing-library swift-testing && [ $0 -eq 0 ] || [ $0 -eq 69 ]'

(By the end of this, you'll have reimplemented a third of SwiftPM. 😬)

Not quite a third, but I'm getting there 🙃