Here We Go

Result Stream

This is a secret technology made by Apple allows you to get real time updates on your test execution process. All you need to do is to pass -resultStreamPath <exiting_file> to xcodebuild and it will start dumping JSON stream right into that file.

We've reverse-engineered models from that file. The results of this job are the ResultStream and ResultStreamModels modules if you want to use it in your projects.

If you want to use it in Emcee though, we've covered it under a FEATURE TOGGLE — that's right, you'll have to opt-in (for now):

• Add EMCEE_USE_RESULT_STREAM = true to your test enviornments (via test arg file) and your tests will be executed via this new high technology.

Bugtressing and Speedbumping

• Emcee kickstart command will now correctly start a worker even if one never started before

• JSONStream CPU performance improved by 3.5x (or 350%). Also, we now use bytes (UInt8) instead of unicode scalars. Only UTF8 encoding is supported though.

It is never too late to reduce redundant logging

• Remove annoying error logs of job deletion error

It is never too late to expose TestsWorkingDirectorySupport target for public needs

• Exposed TestsWorkingDirectorySupport target. Use it by depending on EmceeInterfaces

Useless

Metrics

• New metric! Useless xcodebuild invocations. This event describes a situation when Emcee invokes xcodebuild to run the tests, but it never runs them, instead waits for something (e.g. testmanagerd) and then terminates. Usually this happens when too much simulators are being in use on a machine. Reducing simulator count or rebooting your machine may help with reducing useless invocations.

• Fixed dequeud/enqueued metric values

• New metric reporting system! Emcee can now report to Statsd! Analytics configuration accepts new settings for it.

Bugtressing

• When you enqueue tests with non-satisfiable worker requirements, these tests will be marked as failed if no worker is available to run these tests instead of allowing queue to keep them enqueued infinitely while waiting for suitable worker to appear. 🦺

• Fixed running Emcee on larger machine farms with more than 64 computers. 🤡

• Scheduling tests with empty worker requirements now again supported. 🦵🏻

APIs

• [Synchronous]QueueClient classes have finally been deleted.

• JobStateFetcher for fetching current job state, JobResultsFetcher for fetching job results, JobDeleter for deleting job are the new APIs of QueueClient module.

Reduce

Worker Sharing is ON

By default. Read more about this feature here.

Worker Capabilties

New feature! Allows to schedule tests to workers that do meet specific requirements. Read more about this feature here. Currently only available Xcode versions are exposed, but this can be extended. Submit your ideas.

Kickstarting Silent Workers

New feature! Allows you to send a kickstart command to the queue via REST to make it attempt revive the specific workers. Useful if you kill EmceeWorker process for machine maintenance and then need to get it back to the working state.

Metrics

All metrics now have emcee version in them

Cleaning up

• Lots of fields in test arg file now optional and have default values. The shortest valid test arg file can be very small now:

{
    "jobId": "jobId",
    "entries": [
    {
        "testsToRun": ["all"],
        "testDestination": {"deviceType": "iPhone X", "runtime": "11.3"},
        "testType": "uiTest",
        "buildArtifacts": {
            "appBundle": "http://example.com/App.zip#MyApp/MyApp.app",
            "runner": "http://example.com/App.zip#Tests/UITests-Runner.app",
            "xcTestBundle": "http://example.com/App.zip#Tests/UITests-Runner.app/PlugIns/UITests.xctest"
        }
    }
    ]
}

• Made --emcee-version non optional - this is generated automatically when you do make build.

• Removed --queue-server-destination command line arg from runTestsOnRemoteQueue command. This file has been merged into --queue-server-configuration. By the way, --queue-server-run-configuration became --queue-server-configuration (no stupid run anymore).

• --job-id, --job-group-id, --priority, --job-group-priority arguments has been removed. They all migrated to a test arg file.

• Models target has been removed from Swift package.

Fixes

Demangling buffer size (this is related to test discovery) has been increased from 1K to 10K. Should be "enough for everyone". If it fires, let us know, we can implement dynamic buffer size.

Side Project

EmceeAdmin allows you to observe and control your Emcee workers via GUI. The latest release supports matching Emcee workers and TeamCity agents.

Publicity

APIs

• We've exposed EmceeCommunications library, a set of APIs to discover and talk to Emcee queue.

Other

• enableWorker command correctly enables worker now

• Emcee cache size is now limited to 20Gb by default. In your plugins, if you use cache, set this value to 20Gb as well.

• Log files are cleaned every 3 hours instead of cleaning them on each Emcee invocation.

Cold Fix

• Improve log contents for better readability and usefulness

• Fix partial data parsing from test runner

How It Should Have Been Done

Simulator Settings are Working Correctly 😥

This feature has been implemented for fbxctest previously, and didn't work for xcodebuild test runner. Now this has changed! Emcee correctly patches simulator plist files, and restart affected daemons to ensure patched settings are applied. There is even a new keyboard setting for you to patch - didShowContinuousPathIntroduction, which allows to skip annoying gesture-based input hint.

Synced Simulator State 🚉

Emcee now gets simulator for its actual state instead of keeping it in memory. This allows to avoid some nasty bugs, e.g. when Emcee attempts to shutdown simulator, but the operation takes too long, so Emcee kills xcrun simctl shutdown process in order to proceed, but actually simulator is eventully getting torn down by the Apple mechanics. This led to inconsistent state; not anymore.

New Metrics! 🐎

• test.between_tests.duration - provides a duration between "test has finished" and "test started" events in the same bucket.
• simulator.allocation.duration - provides a duration of how long simulator gets prepared to be used for testing
• test.preflight - provides a duration of how long test runner (e.g. xcodebuild) takes to prepare itself and start running a first test. More precisely, this is a time from when test runner process starts till the time when first test starts.
• test.postflight - this is a duration between test finish event and the termination of test runner process. Describes situation when test runner process finishes running all tests, but does not terminate, wrapping up its work.

Logging Improvements 📄

• Queue server has less redundant logs
• Logs of all launched subprocesses are stored inside ~/Library/Logs/ru.avito.emcee.logs/ like logs of Emcee itself. This is useful for investigation. Emcee cleans all logs that are older than 30 days when it launches, but you may clean up these logs yourself if you don't need them.
• When running tests using runTestsOnRemoteQueue, Emcee now prints number of enqueued and dequeued tests instead of buckets, as well as what tests each worker is executing. 🔥

Bug Fixing Too 🐞

• Emcee now accepts streamable test failures from XCTestJSON. This is our secret technology, yet to be announced later. ㊙️
• Emcee correctly processed localized xcodebuild output.
• Silent workers can become alive, and the whole process of silent detection has been simplified.
• Old log files are deleted in parallel to speed up Emcee launch.
• Emcee now correctly tracks test timeouts when you use xcodebuild as a test runner.

xcodebuild, disabling & enabling workers

xcodebuild

In our environment Emcee was able to perform our complete set of about ~2k unit, ~1.2k application, and ~600 UI tests using pure xcodebuild test runner. This means you can stop providing fbxctest to Emcee and use "testRunnerTool": {"type": "xcodebuild"} in test arg file.

This is still in beta though: errors are being parsed from xcodebuild output, line by line. If error has multiline description, only the first line will be captured by Emcee. We are aiming to fix this in future releases of Emcee.

If you use xcodebuild with Emcee, you have to use insideUserLibrary location for simulators.

Enabling and Disabling Workers on the Fly

Emcee now has two new commands that allow you to disable specified worker, and to re-enable it later. This is useful for maintenance purposes.

$ Emcee disableWorker --worker-id "worker-machine-01" --queue-server "queue-host:41000"
$ Emcee enableWorker --worker-id "worker-machine-01" --queue-server "queue-host:41000"

There is also a corresponding REST handlers which queue server accepts.

Generation of Package.swift File

Package.swift for highly modularized project quickly becomes a mess. Now this file is generated when you invoke make open and make gen. New changes to it's contents should be applied to Package.swift.template.

Xcode 11.4

can finally be used to develop Emcee.

Race condition fix

After performing test discovery, Emcee issues a request to store the obtained result into remote cache. Previously it didn't wait for it to complete. Sometimes this led to request not being completed. Now Emcee waits for request to finish.