Add support for Xcode 11 new xcresult format

[joshdholtz]

Fixes #32

Motivation

Support Xcode 11 xcresult format

Progress

• Allowed reading of new Xcode 11 xcresult file
• Run xcrun commands to get the JSON output
• Run on small sample project to get mapping of ☝️ to match Xcode 10 junit file
• Clean up logic from ☝️
• Link up failure files and line number and error messages
• Test on bigger test suites
• Write tests

Description

Files changed

• Edited test_parser.rb
• Added xcresult.rb
• The majority of the 79 files changed here by adding an .xcresult file for tests

Summary

• Added searching of path for *.xcresult
• Created all the models we needed based off of xcrun xcresulttool formatDescription in xcresult.rb
• Gets JSON formatted results with xcrun xcresulttool get --format json --path #{path}
  • Used to get an id of all the test summaries
  • Also used to get list of all the failures
• Uses ids of test summaries to get specific info for each test summary with xcrun xcresulttool get --format json --path #{path} --id #{id}
  • Used to get list, status, and duration of all tests run
  • Maps failures based upon target name and test name

Results

From Xcode 10

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="7" failures="2">
    <testsuite name="TestUITests" tests="1" failures="0" time="17.92351496219635">
        <testcase classname="TestUITests" name="testExample()" time="17.92015504837036">
        </testcase>
    </testsuite>
    <testsuite name="TestThisDude" tests="6" failures="2" time="0.7454169988632202">
        <testcase classname="TestTests" name="testExample()" time="0.0010919570922851562">
        </testcase>
        <testcase classname="TestTests" name="testFailureJosh1()" time="0.006746053695678711">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestTests/TestTests.swift:37)">
            </failure>
        </testcase>
        <testcase classname="TestTests" name="testPerformanceExample()" time="0.4788789749145508">
        </testcase>
        <testcase classname="TestThisDude" name="testExample()" time="0.00041794776916503906">
        </testcase>
        <testcase classname="TestThisDude" name="testFailureJosh2()" time="0.0015239715576171875">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestThisDude/TestThisDude.swift:36)">
            </failure>
        </testcase>
        <testcase classname="TestThisDude" name="testPerformanceExample()" time="0.2513129711151123">
        </testcase>
    </testsuite>
</testsuites>

From Xcode 11

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="7" failures="2">
    <testsuite name="TestUITests" tests="1" failures="0" time="16.05245804786682">
        <testcase classname="TestUITests" name="testExample()" time="16.05245804786682">
        </testcase>
    </testsuite>
    <testsuite name="TestThisDude" tests="6" failures="2" time="0.5279300212860107">
        <testcase classname="TestTests" name="testExample()" time="0.0005381107330322266">
        </testcase>
        <testcase classname="TestTests" name="testFailureJosh1()" time="0.006072044372558594">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestTests/TestTests.swift#CharacterRangeLen=0&amp;EndingLineNumber=36&amp;StartingLineNumber=36)">
            </failure>
        </testcase>
        <testcase classname="TestTests" name="testPerformanceExample()" time="0.2661939859390259">
        </testcase>
        <testcase classname="TestThisDude" name="testExample()" time="0.0004099607467651367">
        </testcase>
        <testcase classname="TestThisDude" name="testFailureJosh2()" time="0.001544952392578125">
            <failure message="XCTAssertTrue failed (/Users/josh/Projects/fastlane/test-ios/TestThisDude/TestThisDude.swift#CharacterRangeLen=0&amp;EndingLineNumber=35&amp;StartingLineNumber=35)">
            </failure>
        </testcase>
        <testcase classname="TestThisDude" name="testPerformanceExample()" time="0.2531709671020508">
        </testcase>
    </testsuite>
</testsuites>

[saschagordner]

Hey josh. First of all, thanks for the PR!
I only had to do one thing getting it to work, if someone else is wondering why it doesn't work initially:
If you've defined the exact path to the result bundle within the trainer action, adjust the path to point only to the directory where the result bundle sits in.
After doing that it worked for me.

There's still an obsolete error message in case one points to an incorrect path:

No test result files found in directory x, make sure the file name ends with 'TestSummaries.plist'

As TestSummaries.plist doesn't exist anymore, could you please update it to a more appropriate error messaage @joshdholtz? Thanks!

[joshdholtz]

@forceunwrap Thanks for mentioning that! I fixed this PR for both of those 😊

[saschagordner]

That was fast, thanks 😁