Proposal for testing support in SwiftPM #51

mxcl · 2015-12-12T00:04:41Z

Our proposal for how testing infrastructure should operate and be implemented in the package manager.

pwightman · 2015-12-12T07:13:42Z

proposals/0008-package-manager-testing.md

+    └── Tests
+        └── TestFoo.swift
+
+The filename: `TestFoo.swift` is arbituary.


kylef · 2015-12-12T10:50:00Z

proposals/0008-package-manager-testing.md

+However, any such decision would warrant extensive design consideration,
+so as to avoid polluting or crowding the command-line interface.
+Should there be sufficient demand and justification for it, though,
+it would be straightforward to add this functionality.


I would really like to see any arguments after swift build --test or swift test pass though to the test runner.

Example, the test framework Spectre allows command line flags to use different reporters. For example, --tap, --dot etc.

Very much agree in terms of command-line arguments being used by test runners, and in terms of the necessity of custom test reporters.

Still, I believe Swift's Process.environment may be able to suffice for now, and would allow us to decouple that important discussion from this proposal.

modocache · 2015-12-12T15:42:48Z

Currently the test suites in swift-corelibs-foundation's and swift-package-manager use different scripts/Xcode projects for testing, but both do the following:

On OS X, Objective-C XCTest is executed via $(DEVELOPER_DIR)/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest. This program gathers up all the test cases and runs them.
On Linux, XCTest is executed via XCTMain(), called from within a main.swift file. Both foundation and package-manager use different systems to ensure this file is not run on Linux.

How will this change as a result of this proposal, if at all?

parkera · 2015-12-12T15:45:01Z

Ultimately we want swift-corelibs-foundation to be able to build, run, and test itself with swiftpm.

modocache · 2015-12-12T15:45:35Z

@parkera On Linux? Or on OS X as well?

parkera · 2015-12-12T15:46:11Z

On Linux - the plan with the core libraries is to provide their features where we do not have the Darwin equivalents.

danthorpe · 2015-12-14T11:24:47Z

proposals/0008-package-manager-testing.md

+a test for "Foo" will depend on compilation of the library target `Foo`.
+Any additional dependencies or dependencies that could not be automatically determined
+would need to be specified in a package manifest separately.
+


To clarify the author's intent for resolving additional dependencies - does this mean that in Package.swift there will be a "private" section for dependencies used by the tests, (things like DVR), or that the Test directory will include its own manifest?

Just want to link this to swiftlang/swift-package-manager#74

There is a pull request and discussion around "private dependencies" in this pull request.

As much as I like the idea of having another manifest and thus keeping it all separate, maintaining two manifests is an extra layer of tedium for packagers, so I think we'll (longterm) be learning towards keeping all of this in one manifest file. The linked PR is being reviewed today, but is not a firm decision, we may at a later time change the manifest format before Swift 3 drops.

pcantrell · 2015-12-14T19:01:32Z

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

ghost · 2015-12-15T04:34:52Z

proposals/0008-package-manager-testing.md

+        ├── Foo.swift
+        └── Tests
+            └── Test.swift
+


Is there a reason for this variant? Will one format be recommended over the other? Having one recommended structure seems like it would be better in terms of consistency/providing guidance to developers.

One will be recommended. I believe in strong recommendations for flexible workflows.

drewcrawford · 2015-12-15T21:30:03Z

How are perf tests handled? Is that:

A separate command that should be proposed and discussed totally separately? swift build --perftest?
A flag to test? swift build --test --performance-only (or swift build --test --with-performance)
Do they run as part of --test?
Do they live in the test folder?

Detail the ability to specify specific tests to run.

jpsim · 2015-12-16T08:17:38Z

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

This is a concern of mine as well. Tests that require to be run on a physical device or on a specific platform would also fall into this category.

Although I think it should be sufficient for the first version of this to allow opting out of certain test cases within this directory structure. Maybe by having a static var on XCTestCase subclasses:

class FooDeviceTests: XCTestCase {
  static var excludeFromPackageManagerTests = true
}

modocache · 2015-12-16T17:59:44Z

(As I commented above) I don't love the idea of coupling swiftpm to corelibs-xctest. Adding something like XCTestCaseProvider.excludeFromPackageManagerTests would then also couple corelibs-xctest to concepts that only exist in swiftpm, which I think is undesirable.

Theoretically, you could exclude certain test cases from swiftpm using existing concepts:

class FooDeviceTests: XCTestCase {
    var allTests: [(String -> () -> Void)] {
        var tests = [(String -> () -> Void)]()
        if !Process.arguments.contains("swift-build") {
            tests.append(("test_onPhysicalDevice", test_onPhysicalDevice))
        }
        return tests
    }
}

Going forward, we should have test frameworks like swift-corelibs-xctest and Spectre determine how to aggregate tests. When left to implement their own aggregation schemes, testing frameworks can innovate: the way JUnit 4 implements aggregation is different from how RSpec uses test filters; both have pros and cons. Asserting too much control in how tests are run in the package manager stifles these frameworks' opportunity to develop new concepts.

The command-line arguments part of the proposal, introduced in 68e618c, is great. It sounds like it would make filtering easy to implement, either in swift-corelibs-xctest or other frameworks:

// swift build -t --exclude broken

// MyTestingFramework parses command-line arguments to
// determine which tests to run or not run.
class MyBrokenTests: MyTestingFrameworkTestCase {
    // These tests don't work yet, so I tag them as 'broken'.
    var tags: [String] { ['broken'] }
    func test_myTestThatDoesntPassYet() { /* ... */ }    
}

mxcl · 2015-12-16T18:50:08Z

Currently the test suites in swift-corelibs-foundation's and swift-package-manager use different scripts/Xcode projects for testing, but both do the following:

On OS X, Objective-C XCTest is executed via $(DEVELOPER_DIR)/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest. This program gathers up all the test cases and runs them.

On Linux, XCTest is executed via XCTMain(), called from within a main.swift file. Both foundation and package-manager use different systems to ensure this file is not run on Linux.

How will this change as a result of this proposal, if at all?

Long term a goal is to remove the divergence here, but there is no timeline for it.

mxcl · 2015-12-16T18:57:32Z

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

There is no proposal for this at this time. However, we intend to support other test frameworks that conform to a protocol vended by swiftpm, it seems easy enough to design the protocol with these considerations in mind (it could signify the end of a test with a closure to allow asynchronous running mechanisms, and start-end functions could be provided to allow control of external test processes).

mxcl · 2015-12-16T18:59:25Z

How are perf tests handled? Is that:

A separate command that should be proposed and discussed totally separately? swift build --perftest?

From the top of my head, I don’t see why it would need to be separated.

A flag to test? swift build --test --performance-only (or swift build --test --with-performance)

We could certainly allow this kind of specification. Currently the proposal says it will accept the name of the test case on the command line and that seems sufficient initially.

Do they run as part of --test?

I would think: yes.

Do they live in the test folder?

I would think: yes.

mxcl · 2015-12-16T19:00:36Z

Although I think it should be sufficient for the first version of this to allow opting out of certain test cases within this directory structure. Maybe by having a static var on XCTestCase subclasses:
class FooDeviceTests: XCTestCase {
  static var excludeFromPackageManagerTests = true
}

It seems to me that #ifs in the Package.swift would be a better way to exclude targets for different target platforms.

mxcl · 2015-12-16T19:02:38Z

Going forward, we should have test frameworks like swift-corelibs-xctest and Spectre determine how to aggregate tests. When left to implement their own aggregation schemes, testing frameworks can innovate: the way JUnit 4 implements aggregation is different from how RSpec uses test filters; both have pros and cons. Asserting too much control in how tests are run in the package manager stifles these frameworks' opportunity to develop new concepts.

If you feel we are missing any understandings here, please feel free to enlighten us. I am certainly less of an expert in this area than I'd like.

DougGregor · 2015-12-16T19:07:48Z

Can we take this discussion over to a mailing list? Presumable the package manager mailing list, since it's not really in the purview of the general swift-evolution list?

modocache · 2015-12-16T19:13:58Z

If you feel we are missing any understandings here, please feel free to enlighten us.

Oops, sorry if my comment came across as sanctimonious here! 😝

My intention was simply to point out that baking into the package manager the ability to include or exclude performance tests, depending on how that was implemented, may make it harder for test frameworks to build cool new features. I don't think anyone's missing any understandings; based on your comments above I think we're all on the same page! 👍

Can we take this discussion over to a mailing list?

Sounds fine to me!

pcantrell · 2015-12-16T19:22:28Z

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

There is no proposal for this at this time. However, we intend to support other test frameworks that conform to a protocol vended by swiftpm, it seems easy enough to design the protocol with these considerations in mind

I’m less concerned about the details of how the test framework is invoked, and more about being about to put all the things in Tests that belong there, without worrying about that tripping up SwiftPM. That means:

being able to have multiple modules under Tests, some of which are not run by SwiftPM by default,
having the ability to run those non-default test modules via SwiftPM, and
having the ability to put directory structures under Tests that SwiftPM does not know how to deal with at all, and run through some different mechanism altogether.

It’s unclear to me from the wording of the current proposal whether this is covered. It sounds like SwiftPM would find everything named *.swift (or maybe *Test.swift) under the test directory and run it as a test?

mxcl · 2015-12-16T19:31:16Z

If anyone has any further comments then we should take them to swift-build-dev@swift.org. I will finish by replying to the last two comments. The intent for comments here is more to correct things, it was my mistake to direct comments here initially.

Oops, sorry if my comment came across as sanctimonious here! 😝

Not at all, I seek merely to ensure we have considered all aspects.

I’m less concerned about the details of how the test framework is invoked, and more about being about to put all the things in Tests that belong there, without worrying about that tripping up SwiftPM. That means:

being able to have multiple modules under Tests, some of which are not run by SwiftPM by default,

having the ability to run those non-default test modules via SwiftPM, and

having the ability to put directory structures under Tests that SwiftPM does not know how to deal with at all, and run through some different mechanism altogether.
It’s unclear to me from the wording of the current proposal whether this is covered. It sounds like SwiftPM would find everything named *.swift (or maybe *Test.swift) under the test directory and run it as a test?

Excluding targets will be possible as a result of future work on Package.swift.

Proposal for testing support in SwiftPM

Structured concurrency: discuss what executors tasks start on

Proposal for testing support in SwiftPM

e259d37

pwightman reviewed Dec 12, 2015
View reviewed changes

Spelling correction

ca7c2af

kylef reviewed Dec 12, 2015
View reviewed changes

danthorpe reviewed Dec 14, 2015
View reviewed changes

kylef mentioned this pull request Dec 14, 2015

Add Private dependency to Manifest swiftlang/swift-package-manager#74

Merged

ghost reviewed Dec 15, 2015
View reviewed changes

Command Line Interface

68e618c

Detail the ability to specify specific tests to run.

Clarify that multiple test-modules can be created

dc2a3de

Add CLI flag to build without tests

9aa76cc

keith mentioned this pull request Dec 17, 2015

Swift Package Manager support lyft/mapper#16

Closed

rballard added a commit that referenced this pull request Jan 2, 2016

Merge pull request #51 from mxcl/swiftpm-testing

58adcc1

Proposal for testing support in SwiftPM

rballard merged commit 58adcc1 into swiftlang:master Jan 2, 2016

czechboy0 mentioned this pull request Feb 14, 2016

Migrate to swiftpm-provided XCTest vapor/redis#9

Closed

mxcl deleted the swiftpm-testing branch March 15, 2016 20:07

modocache mentioned this pull request May 19, 2016

Fix building on OSX (script, installation) swiftlang/swift-corelibs-foundation#374

Closed

jckarter pushed a commit that referenced this pull request Feb 5, 2021

Merge pull request #51 from jckarter/structured-concurrency-executor

3512f57

Structured concurrency: discuss what executors tasks start on

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal for testing support in SwiftPM #51

Proposal for testing support in SwiftPM #51

mxcl commented Dec 12, 2015

pwightman Dec 12, 2015

ReadmeCritic Dec 12, 2015

kylef Dec 12, 2015

modocache Dec 12, 2015

modocache commented Dec 12, 2015

parkera commented Dec 12, 2015

modocache commented Dec 12, 2015

parkera commented Dec 12, 2015

danthorpe Dec 14, 2015

kylef Dec 14, 2015

mxcl Dec 16, 2015

pcantrell commented Dec 14, 2015

ghost Dec 15, 2015

mxcl Dec 15, 2015

drewcrawford commented Dec 15, 2015

jpsim commented Dec 16, 2015

modocache commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

DougGregor commented Dec 16, 2015

modocache commented Dec 16, 2015

pcantrell commented Dec 16, 2015

mxcl commented Dec 16, 2015

Proposal for testing support in SwiftPM #51

Proposal for testing support in SwiftPM #51

Conversation

mxcl commented Dec 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

modocache commented Dec 12, 2015

parkera commented Dec 12, 2015

modocache commented Dec 12, 2015

parkera commented Dec 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcantrell commented Dec 14, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewcrawford commented Dec 15, 2015

jpsim commented Dec 16, 2015

modocache commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

mxcl commented Dec 16, 2015

DougGregor commented Dec 16, 2015

modocache commented Dec 16, 2015

pcantrell commented Dec 16, 2015

mxcl commented Dec 16, 2015