-
Notifications
You must be signed in to change notification settings - Fork 314
Moai SDK Automated Testing
As of this writing, our immediate pain point is lack of functional testing. Unit testing can be implemented once the functional testing framework is in-place and operation.
For functional testing of Moai, we will build a custom framework. The short term goal is to test features on Desktop (Windows, OSX and Linux). Once this portion of the framework is complete we will move on to testing features on device (Android, iPhone). While not giving us complete coverage, these tests will be a good first-line defense against our most typical regressions. They will largely replace the code currently help in the 'samples' folder and allow for that folder to be cleaned up and replaced with... samples.
Moai functional tests are Lua scripts that are intended to test the features of the Lua SDK. Each test may be run in two modes: the first mode is called ‘staging’ and the second ‘testing.’ The purpose of the staging step is to produce any device-specific artifacts required to perform the test. For example, a test meant to detect regressions in Moai’s text layout system might set up a text box and do a screen capture when run in staging mode. When run in testing mode, the test would do the same setup but finish with a pixel-compare against the screen capture generated during staging.
Staging and testing artifacts should not be checked into the repository as there may be platform-specific nuances.
To aid in managing the suite of functional tests, we'd like a minimalist web interface for viewing them. The interface should:
- Allow users to browse and visually inspect the test artifacts, in particular for staging. For visual tests this will allow users that the staging artifacts were generated correctly. For example, if the staging artifact is a .png showing a layout or effect then the user may view that .png in the browser to make sure it was rendered correctly.
- Allow users to flag tests to re-generate staging artifacts. This will be necessary if SDK functionality is changed by design. For example, if there is a bug fix to text kerning that invalidates previous .pngs output by test staging.