-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Automated UI Tests
This wiki is to describe the state of test automation in Firefox for iOS.
In main branch, every check in is verified with unit test suite upon merge. It is also enabled by default on Bitrise when a build is triggered via a PR request. However, UI/XCUI tests are scheduled to execute on main branch, and on release branches, once per day due to its size. It takes about 1 hour to complete on Bitrise.
The tests are listed in XCode under the Test navigator tab. Tests that are not enabled for the selected scheme would be greyed out.
Currently, there are four schemes executed on Bitrise for running the tests. They run on a daily basis on master and the production branch thanks to a scheduled build:
- Fennec_Enterprise_UITests scheme for UI Testing.
- Fennec_Enterprise_XCUITest scheme for XCUITesting. This scheme runs around 170-190 tests. It runs once for iPhone and once for iPad every day.
- Fennec_Enterprise_SmokeXCUITest. This scheme has a set of tests that runs daily as part of a bigger Smoketest suite where the rest of the tests can't be automated.
- Fennec_Enterprise_XCUITests_Integration. This schema runs for every PR thanks to an environment set with Jenkins. It only runs a few tests which involve Firefox-iOS and Nightly Desktop. The aim of this is to test that the Sync works between platforms, that's why its name.
These schemes can be also used locally to execute tests on simulators. Fennec_Enterprise_UITests scheme has a additional library (Namely, KIFTest) which is required to run tests in UITests package. For Fennec_Enterprise_XCUITests schemes MappaMundi library is required.
Because Fennec_Enterprise_UITests scheme requires a separate license, XCode will complain if one tries to run the test on the actual device. One workaround is to enable UITests/XCUITests in Fennec scheme, as it does not require the enterprise provisioning profile in the Client setting. However, to run UITests, one must include additional parameters in the Test section of the scheme.
- On 'Arguments Passed On Launch', add 'FIREFOX_CLEAR_PROFILE' and 'FIREFOX_TEST'
Tests under UITests package are considered as 'grey-box' tests, where tests can inject specific configuration or values into the target app prior to testing. It uses KIFTest.
- KIFTest link: https://github.com/kif-framework/KIF If you need to modify the app's settings in order to run tests, then you'll need to write UITest
One downside of UITests is that the app does not reset between tests - the same instance of Firefox iOS is used for the entire duration of the test execution. So each test has their own setup/cleanup method. So far it appears there is no easy way to reset the app between tests, because this would make the running db to lose connection, and cause failures. Any solution to this issue will be welcomed.
One another thing is that because the tests technically execute as unit tests, one cannot pause the execution and manipulate the UI of the app. The UI will be frozen, and it will interact only when the test is running.
Tests under this package are considered complete black-box tests, with no visibility to the innards of the Firefox iOS app state. This is also the test suite that takes most time, with slightly over an hour of execution time. There are many helpful websites on writing XCUITests, like this one. If you need to write a black-box UI test, then you'll need to write XCUITests
One downside of XCUITests is that it appears the SimplePageServer (to prop up mock websites) does not start properly when the XCUITest script calls them. The workaround for this has been the use of GCDWebServers. Thanks to that is possible to use testing websites. There are several examples already implemented. A quick how to use it would be:
- Create your testing website in the correct folder, test-fixtures folder
- Add the test name where the server is implemented
- Open your website as a regular one:
navigator.openURL(path(forTestPage: "testing-page.html"))ornavigator.openNewURL(urlString: path(forTestPage: "testing-page.html"))
When launching these tests the app will start from a clean state, like a fresh install everytime. There are some ways to change this behaviour in order to have data pre-loaded or to start from a different initial state:
So far, prepopulating the History and Bookmarks is possible by loading the Browser DB created while browsing into the app. This DB is created and stored in real devices or simulators and can be found in a similar path:
~/Library/Developer/CoreSimulator/Devices/<DeviceID>/data/Containers/Shared/AppGroup/<AppGroupID>/profile.profile/browser.db
Then the DB file has to be copied to the test-fixtures folder. Now it could be used by any existing or new XCUITest:
func testWithPrepopulatedDB() {
let file = "NameYourDB-browser.db"
let arg = LaunchArguments.LoadDatabasePrefix + file
app.terminate()
restart(app, args: [LaunchArguments.SkipIntro, LaunchArguments.SkipWhatsNew, arg])
// Now there will be items in the History and/or Bookmarks
...
}
The ability to store and access the generated DB could be very useful for future tests and different tests scenarios. For example, it is possible to create a script which runs different tests and manages the DB so that it can be used for others. Please see video example here
- TestA is launched. It creates a browser DB.
- DB is saved and stored correctly.
- TestB is launched. It uses the previous created DB.
From here many more options are available for testing.
Further work to be able to get different prepopulated data, like Logins DB is ongoing
Testing with a specific experiment enabled is possible by loading an experiment file using launchArguments. This can be done for a whole test class or for a single test.
First a json file with the experiment data for the test needs to be created and saved in the folder firefox-ios/Client/Nimbus/TestData. The data for the experiment can be extrapolated from the experiments found in firefox-ios/nimbus-features. For the json the feature name is not needed - only the variables and their values are important here. For example for a UITests that should test with a disabled site reporting feature the following json would be saved for the feature general-app-features :
{
"report-site-issue": {
"status": false
}
}
To load the experiment during testing there is two options:
- Load it only for a specific test
- Load it in a test class so all tests in this class use the same experiment
This only works if the function setUpApp() doesn't launch the app as the experiment has to loaded using launch arguments before the app lauches. Each test case will have to call app.launch() in this case. Before that the test data file and the experiment feature name have to be set in the launch arguments:
var launchArgs = app.launchArguments + ["\(LaunchArguments.LoadExperiment)reportSiteIssueOff"]
launchArgs = launchArgs + ["\(LaunchArguments.ExperimentFeatureName)general-app-features"]
app.launchArguments = launchArgs
app.launch()
In order to use the same experiment for all tests in a test class the experiment will be set up during setup of the test class (in setUpApp()). Here the test data file and the experiment feature name have to be set in the launch arguments. Only after that is done the app can be launched by calling app.launch().
override func setUpApp() {
var launchArgs = app.launchArguments + ["\(LaunchArguments.LoadExperiment)reportSiteIssueOff"]
launchArgs = launchArgs + ["\(LaunchArguments.ExperimentFeatureName)general-app-features"]
app.launchArguments = launchArgs
app.launch()
}
In order to be able to change the initial state of the app is possible to use arguments when launching the app. There are already some defined here. Using these arguments allows us to, for example, skip the intro or clear profile. Check how the defined ones work. In addition to those, more arguments can be added for example to start the app with a different setting by default, using one of the already defined PrefsKeys or adding a new one.
This deserves a separate wiki page. It is strongly recommended to use FxScreenGraph when writing a XCUITest.
The UI/XCUI Test automation status is being updated here: https://github.com/mozilla-mobile/firefox-ios/projects/2. You are welcome to take one from our to-do lists and make contributions.
In order to create new tests these are good practices to follow:
- Have a clear goal of the test
- Make it simply, breaking it into clear steps
- The best approach would be to create one test for one major task
- Common scenarios. Do not try to create tests for edge cases
- Basic scenarios with as simple as possible scripts
- Try to minimize the external dependencies
- Short tests are more likely to be stable
- General cases, if tests only apply for one device has to be clearly specified
Please send questions to irios@mozilla.com or ping :irios in mozilla slack.