-
Notifications
You must be signed in to change notification settings - Fork 582
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for tests executed repeatedly #585
Add support for tests executed repeatedly #585
Conversation
422288f
to
44b3736
Compare
I have two questions regarding this
|
aad695f
to
cae4380
Compare
Yes. TestObserver is called after test case is executed so the feature of counting multiple assertSnapshot calls works as expected.
I've never used Quick or Nimble. Would you be so kind to check with my forked version if it works with your project? I've already checked with our where we have planty of different scenarios but definitely would be cool to have confirmation also from someone else. |
@krzysztofpawski I tried it out and it still produces many snapshots. The way |
Maybe it will make sense to make it unique based on the test line. The only thing that would break it is if someone adds two assertions on the same line |
cae4380
to
922eee3
Compare
@tahirmt could you check it one more time? I found one more case when it was producing too many reference files in our repo. |
@krzysztofpawski I created a demo repo to test this using your fork https://github.com/tahirmt/test-toeventually-behavior-snapshots. |
@tahirmt I'm afraid it can't be fixed in |
That's what I was thinking too. I was thinking of many alternatives but I can't think of a better solution for now. Thanks for looking into it though. |
@stephencelis any possibility to review this PR? |
This includes [this fix](pointfreeco/swift-snapshot-testing#585) which should solve some of the flaky test failures we've been seeing lately, which made test retries also fail. We couldn't update to 1.10.0 because it dropped support for iOS 12.0, but that was fixed in pointfreeco/swift-snapshot-testing#698
This includes [this fix](pointfreeco/swift-snapshot-testing#585) which should solve some of the flaky test failures we've been seeing lately, which made test retries also fail. We couldn't update to 1.10.0 because it dropped support for iOS 12.0, but that was fixed in pointfreeco/swift-snapshot-testing#698 This also reverts #2426, since that's no longer required now that retries should work properly.
Motivation
Xcode 13
introduced new parameters toxcodebuild
that allow to run tests repeatedly (wwdc talk).Unfortunately
SnapshotTesting
doesn't handle such situation in a correct way.Proposal
We can use
XCTestObservation
in order to clean counter map between different test cases.Resolved issues
Closes #577