[SWT-NNNN] Introduce API allowing traits to customize test execution #733
Conversation
stmontgomery
added
documentation
stmontgomery
requested review from
grynspan,
dennisweissmann,
briancroom,
SeanROlszewski and
suzannaratcliff
as code owners
stmontgomery
force-pushed
the
publicize-CustomExecutionTrait
branch
from
75975cf to
75dd9b6
Compare
grynspan
changed the title
|
@swift-ci please test Windows |
| struct MockAPICredentialsTrait: TestTrait, CustomTestExecuting { | ||
| func execute(_ function: @Sendable () async throws -> Void, for test: Test, testCase: Test.Case?) async throws { | ||
| let mockCredentials = APICredentials(apiKey: "...") | ||
| try await APICredentials.$current.withValue(mockCredentials) { |
There was a problem hiding this comment.
We think task locals are going to be a particularly common use case, right? Perhaps we should build out a trait type that sets a task local:
@Test(.withValue(123, for: $foo)) func f() { ... }Or even if we don't make it a trait, can we make it easier to write a trait that sets a task local with minimal boilerplate?
There was a problem hiding this comment.
Agree that would be a very useful, built-in trait to have. Let's consider that separately, since it would ultimately leverage the API proposed here
There was a problem hiding this comment.
Doesn't need to be part of this proposal, but I think it does affect this proposal. Might be worth coming up with an example of when you'd use this stuff other than to set a task local.
There was a problem hiding this comment.
Sure. Briefly: not all work before/after a test is simply mutating state in that process. It may involve writing to the filesystem, or interacting with other system-wide resources. (Clearly, parallelism would need to be considered in such scenarios.)
There was a problem hiding this comment.
Also: Users can still use this facility to mutate global or static state, as long as it complies with Swift concurrency rules
There was a problem hiding this comment.
Can we make sure to add this example to the discussion?
There was a problem hiding this comment.
Where, specifically, do you recommend we place this example code snippet?
|
@stmontgomery Per our conversation offline you mentioned that order of execution of custom execution traits is well-defined to be outer-to-inner, left-to-right -- it would be great if we could explicitly add a note about that in the documentation so that folks are aware of the expected behavior! |
…e per scope, by default
I've added language specifying this ordering to the updated proposal and PR |
… the base verb instead of "execute"
stmontgomery
force-pushed
the
publicize-CustomExecutionTrait
branch
from
fb4f00e to
c53220e
Compare
|
@swift-ci please test |
| suite type (including `init`, `deinit`, and the test function itself) can access | ||
| `self`, so this would grant traits applied to an instance test method access to | ||
| the instance as well. This is certainly interesting, but poses several technical | ||
| challenges that puts it out of scope of this proposal. |
There was a problem hiding this comment.
Mind adding a quick note about a task-local trait here? Danke.
| @@ -72,43 +192,14 @@ extension Trait { | |||
| } | |||
| } | |||
|
|
|||
| extension Trait where TestScopeProvider == Never { | |||
There was a problem hiding this comment.
These extensions will show up (at least minimally) in DocC, so we might want to consider formally documenting them with full markup since they have some interesting behaviours.
(Not a blocker for the PR, just noting.)
| } | ||
|
|
||
| extension SuiteTrait where Self: TestScoping { | ||
| // If `test` is a suite, returns `nil` if `isRecursive` is `true`, else `self`. |
There was a problem hiding this comment.
The comments here literally describe the current implementations but don't really explain why they are what they are. Why does this function return nil if isRecursive is true? etc.
This includes an API proposal and code changes to introduce new API for custom traits to customize test execution.
View the API proposal for more details.
Motivation:
One of the primary motivations for the trait system in Swift Testing, as described in the vision document, is to provide a way to customize the behavior of tests which have things in common. If all the tests in a given suite type need the same custom behavior,
initand/ordeinit(if applicable) can be used today. But if only some of the tests in a suite need custom behavior, or tests across different levels of the suite hierarchy need it, traits would be a good place to encapsulate common logic since they can be applied granularly per-test or per-suite. This aspect of the vision for traits hasn't been realized yet, though: theTraitprotocol does not offer a way for a trait to customize the execution of the tests or suites it's applied to.Customizing a test's behavior typically means running code either before or after it runs, or both. Consolidating common set-up and tear-down logic allows each test function to be more succinct with less repetitive boilerplate so it can focus on what makes it unique.
Checklist: