Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#185 #189 process termination code #190

Merged
merged 3 commits into from
Nov 5, 2020
Merged

#185 #189 process termination code #190

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f29c8b4
Use a defer to guard against future early returns.
ZevEisenberg Nov 2, 2020
89a140c
Detect process exit code instead of relying just on log parsing.
ZevEisenberg Nov 2, 2020
80bb3f4
Fix reporter method
rakaramos Nov 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Sources/muterCore/MutationTesting/MutationTestingIODelegate.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,16 @@ struct MutationTestingDelegate: MutationTestingIODelegate {
savingResultsIntoFileNamed fileName: String) -> (outcome: TestSuiteOutcome, testLog: String) {
do {
let (testProcessFileHandle, testLogUrl) = try fileHandle(for: fileName)
defer { testProcessFileHandle.closeFile() }
Copy link
Member

@rakaramos rakaramos Nov 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the same behaviour as we had before? It seems now that the file handle is closed after we attempt to read it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defer happens right before the scope closes. It should be the same behavior as before, but now it'll still happen if we add an early return. I was originally going to use a guard in an earlier version of this PR. I ended up not doing it, but I still like using defer for open/close patterns.


let process = try testProcess(with: configuration, and: testProcessFileHandle)
try process.run()
process.waitUntilExit()
testProcessFileHandle.closeFile()

let contents = try String(contentsOf: testLogUrl)

return (
outcome: TestSuiteOutcome.from(testLog: contents),
outcome: TestSuiteOutcome.from(testLog: contents, terminationStatus: process.terminationStatus),
testLog: contents
)

Expand Down
8 changes: 4 additions & 4 deletions Sources/muterCore/TestReporting/TestSuiteOutcome.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ public enum TestSuiteOutcome: String, Codable {
}

extension TestSuiteOutcome {
public static func from(testLog: String) -> TestSuiteOutcome {
public static func from(testLog: String, terminationStatus: Int32) -> TestSuiteOutcome {

if logContainsRuntimeError(testLog) {
if !terminationStatusIsSuccess(terminationStatus) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think regression tests are still passing, right? Because I think xcodebuild returns 65 when tests fails.
I've find this unofficial list of exit codes... if can also improve it later

return .runtimeError
} else if logContainsBuildError(testLog) {
return .buildError
Expand Down Expand Up @@ -53,8 +53,8 @@ extension TestSuiteOutcome {
return try! NSRegularExpression(pattern: "with ([1-9]{1}[0-9]{0,}) failure", options: [])
}

static private func logContainsRuntimeError(_ testLog: String) -> Bool {
return testLog.contains("Fatal error")
static private func terminationStatusIsSuccess(_ terminationStatus: Int32) -> Bool {
return terminationStatus == 0
}

static private func logContainsBuildError(_ testLog: String) -> Bool {
Expand Down
33 changes: 19 additions & 14 deletions Tests/TestReporting/testSuiteResultParsingSpec.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,55 +9,60 @@ class TestSuiteResultParsingSpec: QuickSpec {
context("when a test log doesn't contain a failure, runtime error, or build error") {
it("returns a passed test result") {
var contents = loadLogFile(named: "testRunWithoutFailures_withTestSucceededFooter.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.passed))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.passed))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used 0 for everything except the one case I cared about testing. We should probably think about whether any of the rest of these tests need to change or be expanded.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree (left a comment above b4 reading it, sorry 😄 )


contents = loadLogFile(named: "testRunWithoutFailures_withTestSucceededFooter_buckOutput.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.passed))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.passed))
}
}

context("when a test log contains a failure") {
it("returns a failed test result") {
var contents = loadLogFile(named: "testRunWithFailures_withoutTestFailedFooter.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.failed))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.failed))

contents = loadLogFile(named: "testRunWithFailures_withTestFailedFooter.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.failed))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.failed))

contents = loadLogFile(named: "testRunWithFailures_withTestFailedFooter_singleTestFailure.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.failed))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.failed))

contents = loadLogFile(named: "testRunWithFailures_withTestFailedFooter_noTestFailureCount.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.failed))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.failed))

contents = loadLogFile(named: "testRunWithFailures_withTestFailedFooter_buckOutput.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.failed))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.failed))
}
}

context("when a test log contains a fatal error") {
it("returns a runtime error") {
it("does not return a runtime error if the termination status was 0") {
let contents = loadLogFile(named: "runtimeError_fatalError.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.runtimeError))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.passed))
}
it("returns a runtime error if the termination status was not 0") {
let contents = loadLogFile(named: "runtimeError_fatalError.log")
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: -127)).to(equal(.runtimeError))
}

}

context("when a test log contains a build error") {
it("returns a build error") {
var contents = loadLogFile(named: "buildError_missingProjectFile.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.buildError))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.buildError))

contents = loadLogFile(named: "buildError_runScriptStepFailed.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.buildError))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.buildError))

contents = loadLogFile(named: "buildError_invalidSwiftCode.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.buildError))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.buildError))

contents = loadLogFile(named: "buildError_withTestFailedFooter.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.buildError))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.buildError))

contents = loadLogFile(named: "buildError_buckOutput.log")
expect(TestSuiteOutcome.from(testLog: contents)).to(equal(.buildError))
expect(TestSuiteOutcome.from(testLog: contents, terminationStatus: 0)).to(equal(.buildError))
}
}
}
Expand Down