No tests found when using multi_scan with XCUITest

[craig-jamieson]

New Issue Checklist

• Updated fastlane-plugin-test_center to the latest version
• I read the README.md
• I reviewed the example(s) for the action(s) I am using
• I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
• I have reviewed the Discussions forum to see my question has already been addressed.

Issue Description

I've tried some of the suggestions here, but I'm still encountering the same issue. I'm trying to use multi_scan with a project using xcuitest:

 multi_scan(
        workspace: "project.xcworkspace",
        scheme: "XXXX",
        try_count: 3,
        fail_build: false,
        parallel_testrun_count: 4,
        clean: clean,
        configuration: "S101",
        destination: "platform=iOS Simulator,OS=#{IOS_VERSION},name=#{IPHONE_VERSION}",
        reset_simulator: true,
        include_simulator_logs: false,
        skip_slack: true,
        code_coverage: true,
        only_testing: ["UITests"],
        xcargs: "-jobs `sysctl hw.ncpu | awk '{print 2*$2}'`",
        derived_data_path: DERIVED_DATA_FOLDER,
        output_directory: TEST_RESULTS_FOLDER,
        output_files: "report.html,junit.xml"
        )
    rescue => ex
      report_result = build_failing_tests_report(report_identifier: "Report ID", test_results_folder: TEST_RESULTS_FOLDER)
      total_tests_count = 0
      total_failed_tests = 0
      for failing_test_suite in report_result
        if failing_test_suite[:failures] == "0"
          next
        end

And the result is the next ("Test build Succeeded" but zero tests)

[14:58:32]: ▸ Test build Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+

[14:58:35]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/xxx.app/PlugIns/xxx.xctest'!
[14:58:35]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX?
[14:58:35]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
[14:58:37]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/XXX.app/PlugIns/XXX.xctest'!
[14:58:37]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX?
[14:58:37]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
[14:58:38]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/XXX.app/PlugIns/XXX.xctest'!
[14:58:38]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXXX?
[14:58:38]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
[14:58:38]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/XXXX.app/PlugIns/XXX.xctest'!
[14:58:38]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXX?
[14:58:38]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
[14:58:40]: ----------------------------------------
[14:58:40]: --- Step: build_failing_tests_report ---
[14:58:40]: ----------------------------------------
/Users/distiller/i/test-output/results/junit.xml
+---------------------------+---------------------------------------------------------------+
|                                       Lane Context                                        |
+---------------------------+---------------------------------------------------------------+
| DEFAULT_PLATFORM          | ios                                                           |
| PLATFORM_NAME             | ios                                                           |
| LANE_NAME                 | ios e2e_tests                                                 |
| KEYCHAIN_PATH             | ~/Library/Keychains/path                     |
| ORIGINAL_DEFAULT_KEYCHAIN | "/Users/distiller/Library/Keychains/path" |
+---------------------------+---------------------------------------------------------------+
[14:58:40]: No output file found

+------+----------------------------+-------------+
|                fastlane summary                 |
+------+----------------------------+-------------+
| Step | Action                     | Time (in s) |
+------+----------------------------+-------------+
| 1    | Verifying fastlane version | 0           |
| 2    | default_platform           | 0           |
| 3    | setup_circle_ci            | 0           |
| 4    | ensure_xcode_version       | 2           |
| 💥   | multi_scan                 | 764         |
| 💥   | build_failing_tests_report | 0           |
+------+----------------------------+-------------+

+------------+--------------+----------------+
|          Plugin updates available          |
+------------+--------------+----------------+
| Plugin     | Your Version | Latest Version |
+------------+--------------+----------------+
| versioning | 0.4.5        | 0.5.0          |
+------------+--------------+----------------+
[14:58:40]: To update all plugins, just run
[14:58:40]: $ bundle exec fastlane update_plugins

[14:58:40]: fastlane finished with errors

[!] No output file found

ENABLE_TESTABILITY is enabled.

Summary for scan

+------------------------------------------------+-----------------------------------------------------+
|                                       Summary for scan 2.187.0                                       |
+------------------------------------------------+-----------------------------------------------------+
| workspace                                      | XXX.xcworkspace                                   |
| scheme                                         | XXX Stage                                         |
| fail_build                                     | false                                               |
| clean                                          | false                                               |
| configuration                                  | Stage Debug                                         |
| reset_simulator                                | true                                                |
| skip_slack                                     | true                                                |
| code_coverage                                  | true                                                |
| only_testing                                   | ["UITests"]                                    |
| xcargs                                         | -jobs `sysctl hw.ncpu | awk '{print 2*$2}'`         |
| derived_data_path                              | test-output/derived-data                            |
| output_directory                               | test-output/results                                 |
| output_files                                   | report.html,junit.xml                               |
| output_types                                   | html,junit                                          |
| skip_detect_devices                            | false                                               |
| ensure_devices_found                           | false                                               |
| force_quit_simulator                           | false                                               |
| disable_slide_to_type                          | true                                                |
| reinstall_app                                  | false                                               |
| app_identifier                                 | Prod                                     |
| open_report                                    | false                                               |
| buildlog_path                                  | artifacts/buildlogs/scan                            |
| should_zip_build_products                      | false                                               |
| output_xctestrun                               | false                                               |
| use_clang_report_name                          | false                                               |
| disable_concurrent_testing                     | false                                               |
| build_for_testing                              | true                                                |
| slack_use_webhook_configured_username_and_icon | false                                               |
| slack_username                                 | fastlane                                            |
| slack_icon_url                                 | https://fastlane.tools/assets/img/fastlane_icon.png |
| slack_only_on_failure                          | false                                               |
| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                   |
| skip_package_dependencies_resolution           | false                                               |
| disable_package_automatic_updates              | false                                               |
| use_system_scm                                 | false                                               |
| number_of_retries                              | 0                                                   |
| include_simulator_logs                         | false                                               |
| skip_build                                     | false                                               |
| xcode_path                                     | /Applications/Xcode-12.5.1.app 

Environment

🚫 fastlane environment 🚫
Stack



Key
Value




OS
11.4


Ruby
2.6.4


Bundler?
true


Git
git version 2.26.2


Installation Source
~/ios-app/vendor/cache/ruby/2.6.0/bin/fastlane


Host
macOS 11.4 (20F71)


Ruby Lib Dir
~/.rbenv/versions/2.6.4/lib


OpenSSL Version
OpenSSL 1.1.1d  10 Sep 2019


Is contained
false


Is homebrew
false


Is installed via Fabric.app
false


Xcode Path
/Applications/Xcode.app/Contents/Developer/


Xcode Version
12.5.1


Swift Version
5.4.2



System Locale



Variable
Value





LANG
en_GB.UTF-8
✅


LC_ALL




LANGUAGE





fastlane files:
`./fastlane/Fastfile`
# This file contains the fastlane.tools configuration
# You can find the documentation at https://docs.fastlane.tools
#
# For a list of all available actions, check out
#
#     https://docs.fastlane.tools/actions
#
require 'net/http'
require 'uri'
fastlane_require 'yaml'

fastlane_version "2.188.0"

default_platform(:ios)


platform :ios do
  before_all do
    # Allow up to 6 attempts for starting xcodebuild
    ENV["FASTLANE_XCODEBUILD_SETTINGS_RETRIES"] = "5"

    # Prepare CircleCi configuration, including a usable Keychain
    setup_circle_ci

    # This uses the .xcode-version file in the project directory to ensure we
    # use a consistent Xcode version to ship and on the CI to test.
    # If upgrading to a new Xcode, change the `.xcode-version` file.
    ensure_xcode_version
  end
  
  import("./Fastfiles/Setup.rb")

  import("./Fastfiles/Testing.rb")  

  import("./Fastfiles/Release.rb")  
  
  import("./Fastfiles/EmergencyRelease.rb")    

  import("./Fastfiles/Utilities.rb")      
  
  import("./Fastfiles/Build.rb")

  # CircleCI exclusive lanes. Only imported if IS_CI_MACHINE
  if ENV["IS_CI_MACHINE"] == "TRUE" 
    import("./Fastfiles/CircleCI.rb")
  end  
end

# Path resolvers
def build_output_path
  File.expand_path("../#{BUILD_OUTPUT_FOLDER}")
end

def ipa_file_path
  File.join(build_output_path, IPA_FILENAME)
end

def dsyms_file_path
  File.join(build_output_path, DSYM_FILENAME)
end

`./fastlane/Appfile`
app_identifier "Prod" # The bundle identifier of your app
apple_id "" # Your Apple email address

# For more information about the Appfile, see:
#     https://docs.fastlane.tools/advanced/#appfile

fastlane gems



Gem
Version
Update-Status




fastlane
2.188.0
🚫 Update available



Loaded fastlane plugins:



Plugin
Version
Update-Status




fastlane-plugin-aws_s3
2.0.2
🚫 Update available


fastlane-plugin-firebase_app_distribution
0.2.9
✅ Up-To-Date


fastlane-plugin-versioning
0.4.5
🚫 Update available


fastlane-plugin-test_center
3.15.3
✅ Up-To-Date



Loaded gems



Gem
Version




did_you_mean
1.3.0


bundler
2.1.4


rake
13.0.6


CFPropertyList
3.0.3


concurrent-ruby
1.1.9


i18n
1.8.10


minitest
5.14.4


thread_safe
0.3.6


tzinfo
1.2.9


activesupport
5.2.6


public_suffix
4.0.6


addressable
2.8.0


httpclient
2.8.3


json
2.5.1


algoliasearch
1.27.5


rubyzip
2.3.2


apktools
0.7.4


artifactory
3.0.15


atomos
0.1.3


aws-eventstream
1.1.1


aws-partitions
1.479.0


aws-sigv4
1.2.4


jmespath
1.4.0


aws-sdk-core
3.117.0


aws-sdk-kms
1.44.0


aws-sdk-s3
1.96.2


babosa
1.0.4


claide
1.0.3


colored2
3.1.2


cork
0.3.0


nap
1.1.0


open4
1.3.4


claide-plugins
0.9.2


fuzzy_match
2.0.4


netrc
0.11.0


ffi
1.15.3


ethon
0.14.0


typhoeus
1.4.0


cocoapods-core
1.10.1


cocoapods-deintegrate
1.0.4


cocoapods-downloader
1.4.0


cocoapods-plugins
1.0.0


cocoapods-search
1.0.0


cocoapods-trunk
1.5.0


cocoapods-try
1.2.0


escape
0.0.4


fourflusher
2.3.1


gh_inspector
1.1.3


molinillo
0.6.6


ruby-macho
1.4.0


nanaimo
0.3.0


rexml
3.2.5


xcodeproj
1.20.0


cocoapods
1.10.1


colored
1.2


colorize
0.8.1


highline
2.0.3


commander
4.6.0


faraday-em_http
1.0.0


faraday-em_synchrony
1.0.0


faraday-excon
1.1.0


faraday-httpclient
1.0.1


faraday-net_http
1.0.1


faraday-net_http_persistent
1.2.0


faraday-patron
1.0.0


multipart-post
2.0.0


ruby2_keywords
0.0.5


faraday
1.5.1


faraday-http-cache
2.2.0


rchardet
1.8.0


git
1.8.1


kramdown
2.3.1


kramdown-parser-gfm
1.1.0


no_proxy_fix
0.1.2


sawyer
0.8.2


octokit
4.21.0


unicode-display_width
1.7.0


terminal-table
1.8.0


danger
8.2.3


thor
0.20.3


danger-swiftlint
0.24.5


declarative
0.0.20


digest-crc
0.6.4


unf_ext
0.0.7.7


unf
0.1.4


domain_name
0.5.20190701


dotenv
2.7.6


emoji_regex
3.2.2


excon
0.85.0


http-cookie
1.0.4


faraday-cookie_jar
0.0.7


faraday_middleware
1.0.0


fastimage
2.2.4


jwt
2.2.3


memoist
0.16.2


multi_json
1.15.0


os
1.1.1


signet
0.15.0


googleauth
0.16.2


mini_mime
1.1.0


trailblazer-option
0.1.1


uber
0.1.0


representable
3.1.1


retriable
3.1.2


webrick
1.7.0


google-apis-core
0.4.1


google-apis-androidpublisher_v3
0.9.0


google-apis-playcustomapp_v1
0.5.0


google-apis-iamcredentials_v1
0.6.0


google-apis-storage_v1
0.6.0


google-cloud-env
1.5.0


google-cloud-errors
1.1.0


google-cloud-core
1.6.0


google-cloud-storage
1.34.1


mini_magick
4.11.0


naturally
2.2.1


plist
3.6.0


security
0.1.3


simctl
1.6.8


terminal-notifier
2.0.0


tty-screen
0.8.1


tty-cursor
0.7.1


tty-spinner
0.9.3


word_wrap
1.0.0


rouge
2.0.7


xcpretty
0.3.0


xcpretty-travis-formatter
1.0.1


mime-types-data
3.2021.0704


mime-types
3.3.1


fastlane-plugin-aws_s3
2.0.2


fastlane-plugin-firebase_app_distribution
0.2.9


trainer
0.9.1


xctest_list
1.2.1


fastlane-plugin-test_center
3.15.3


fastlane-plugin-versioning
0.4.5


mini_portile2
2.5.3


racc
1.5.2


nokogiri
1.11.7


xcode-build-times
0.1.1


xcode-install
2.6.2




generated on: 2021-08-20

[lyndsey-ferguson]

Hello @craig-jamieson you didn't explicitly mention it, so I have to ask, did you follow the suggestion to:

[14:58:35]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/xxx.app/PlugIns/xxx.xctest'!
[14:58:35]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX?

?

[craig-jamieson]

Hello @craig-jamieson you didn't explicitly mention it, so I have to ask, did you follow the suggestion to:

[14:58:35]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/xxx.app/PlugIns/xxx.xctest'!
[14:58:35]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX?

?

Hello @lyndsey-ferguson, thanks for the reply

Yes, I've enabled testability for the appropriate test target

[lyndsey-ferguson]

Okay, the next thing I have to ask is for you to run your lane, but this time with the --verbose flag and include the logs.

[craig-jamieson]

Okay, the next thing I have to ask is for you to run your lane, but this time with the --verbose flag and include the logs.

Thanks, see attached
e2e1.odt

[lyndsey-ferguson]

The logs indicate that the plugin is crashing in the runner and that exception is caught in your Fastfile. The logs do not provide enough information to indicate where in the runner it is crashing. I've made some modifications to provide more logs.

Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-353-exception-when-starting-tests"

[craig-jamieson]

Thank you!

See attached
e2et.txt

[lyndsey-ferguson]

For my reference, this is the guts of where this is failing:

DEBUG [2021-08-31 10:09:10.00]: < setup_testcollector
DEBUG [2021-08-31 10:09:10.00]: > setup_logcollection
DEBUG [2021-08-31 10:09:10.00]: < done in TestCenter::Helper::MultiScanManager.initialize
DEBUG [2021-08-31 10:09:10.00]: > remove_preexisting_test_result_bundles
DEBUG [2021-08-31 10:09:10.00]: > remote_preexisting_xcresult_bundles
DEBUG [2021-08-31 10:09:10.00]: > setup_testcollector

FAIL HERE :(

[lyndsey-ferguson]

@craig-jamieson, I just noticed that you have this -jobs sysctl hw.ncpu | awk '{print 2*$2}' for the xcargs parameter. I imagine that you're trying to achieve parallel jobs with this, but it may interfere with the logic that I've built into multi_scan's parallel_testrun_count....just food for thought as I add more logs.

[lyndsey-ferguson]

Can you modify your Pluginfile per my instructions below, remove your Gemfile.lock file, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-353-exception-when-starting-tests"

Please attach the logs. Thanks!

[craig-jamieson]

Hi @lyndsey-ferguson

See logs attached.

[lyndsey-ferguson]

For my own reference, it is failing somewhere here:

DEBUG [2021-09-07 10:06:28.38]: < setup_testcollector
DEBUG [2021-09-07 10:06:28.38]: > setup_logcollection
DEBUG [2021-09-07 10:06:28.38]: < done in TestCenter::Helper::MultiScanManager.initialize
DEBUG [2021-09-07 10:06:28.41]: > remove_preexisting_test_result_bundles
DEBUG [2021-09-07 10:06:28.41]: > remove_preexisting_xcresult_bundles
DEBUG [2021-09-07 10:06:28.41]: About to run_test_batches
DEBUG [2021-09-07 10:06:28.41]: > setup_testcollector
DEBUG [2021-09-07 10:06:28.41]: checking if the platform is :ios_simulator
DEBUG [2021-09-07 10:06:28.41]: working with scan_destinations: ["platform=iOS Simulator,OS=14.5,name=iPhone 11 Pro"]
INFO [2021-09-07 10:06:28.76]: ----------------------------------------
INFO [2021-09-07 10:06:28.76]: --- Step: build_failing_tests_report ---
INFO [2021-09-07 10:06:28.76]: ----------------------------------------

[lyndsey-ferguson]

Okay, so I believe I know what the error is: the :destination is being provided and most of the users do not pass it in directly (or if they do, they provide the udid value). The code at this point is looking for a udid value to identify the unique simulator. The code should detect when it is missing and look it up. I also see that the old way of specifying the simulator with the :device parameter, i.e. "iPhone 11 Pro (14.5)" is now not working with the underlying scan command.

@craig-jamieson for now, do you have a way of getting the udid for the iPhone 11 Pro you want to use and provide that as an additional text in the :destination parameter? ("platform=iOS SImulator,OS=14.5,name=iPhone 11 Pro,udid=")

[craig-jamieson]

Thanks @lyndsey-ferguson for the info
I'm having some trouble using the udid as a parameter at the moment, is there anything else you can suggest as a workaround?

[lyndsey-ferguson]

You should be able to get the udid value via:

xcrun simctl list devices --json

https://www.iosdev.recipes/simctl/

Find the iPhone 11 Pro and copy the udid of the device you want.

If that is already what you are doing, can you describe what the trouble is that you're having? I don't know of another work around at the moment, I will let my subconscious chew on it though.

[lyndsey-ferguson]

@craig-jamieson bump

[craig-jamieson]

@lyndsey-ferguson I made some progress, I was able to add the UDID as a parameter but the tests are still failing to execute, see below excerpt from logs:

[worker 5] INFO [2021-09-13 18:05:54.12]: ▸ ** TEST EXECUTE FAILED **
[worker 5] 2021-09-13 18:05:46.985 xcodebuild[99117:9444605] [MT] IDETestOperationsObserverDebug: 246.121 elapsed -- Testing started completed.
[worker 5] 2021-09-13 18:05:46.985 xcodebuild[99117:9444605] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[worker 5] 2021-09-13 18:05:46.985 xcodebuild[99117:9444605] [MT] IDETestOperationsObserverDebug: 246.121 sec, +246.121 sec -- end
[worker 5] Testing failed:
[worker 5] 	XXXUITests:
[worker 5] 		The test runner encountered an error (Failed to prepare device 'iPhone 12 Clone 5 TestCenter::Helper::MultiScanManager::SimulatorHelper<70349616173520>' for impending launch. If you believe this error represents a bug, please attach the result bundle at /var/folders/12/bp7yqvlj1v70pk6nm_4w17900000gn/T/derived_data_path20210913-96407-1jrvf7q-worker-5/Logs/Test/Test-Transient Testing-2021.09.13_18-01-40-+0100.xcresult. (Underlying Error: Unable to boot the Simulator. launchd failed to respond. (Underlying Error: Failed to start launchd_sim: could not bind to session, launchd_sim may have crashed or quit responding)))
[worker 5] 
[worker 5] ** TEST EXECUTE FAILED **
[worker 5] ERROR [2021-09-13 18:05:54.46]: Exit status: 65
[worker 5] DEBUG [2021-09-13 18:05:54.58]: Batch #5 incrementing retry count to 1
[worker 5] DEBUG [2021-09-13 18:05:54.59]: Scan unable to test for batch #5
[worker 5] ERROR [2021-09-13 18:05:54.59]: 
[worker 5] Error building the application. See the log above.

[lyndsey-ferguson]

Hm. Can you include more logs. Also, show me the line for the device that you found after running:

xcrun simctl list devices --json

🙏

[craig-jamieson]

Hello,

See the full logs attached.

e2et.txt

[lyndsey-ferguson]

I see that the xargs flag that you're passing (-jobs sysctl hw.ncpu | awk '{print 2*$2}'), could be splitting the command line to xcodebuild so that it is not passing the tests to run, can you remove that option and retry the command?

[craig-jamieson]

Hi @lyndsey-ferguson,

Sorry for the delay, the tests are now being executed 🙌
I'm experiencing some other issues with circleci but I'll raise that separately after I've investigated

[lyndsey-ferguson]

Okay, great! I'll put this issue back on the to-do pile now that you're unblocked.

[lyndsey-ferguson]

I want to apologize that I have not responded to this issue. A family health issue is requiring me to narrow my focus on the essentials and I don't have the time I need to focus on the plugin more than a minute. If you can get some other contributors to this project to help with this, I would consider merging a resulting PR.