Autobot overview

Autobot

Status: Alpha

An autobot is a system for performing automated tests that simulate user behavior, thereby testing a large number of subsystems. The objectives are to reduce crashes, identify functional and drawing regressions, and make manual testing easy. Autobot can be used both for writing automated tests that can be run on thousands of files automatically, comparing with a known result (prepared in advance), and for writing scripts that can facilitate manual testing, passing smoke tests, automate routine tasks. Test cases are written in js scripts

Configuration

We supply some scripts with the distribution. They are located in $APPDATA/autobotscripts. We can add self scripts by placing them in $USERDATA/AutobotScripts. We can also set the environment variable MU_AUTOBOT_SCRIPTS_PATH. Then, scripts will be searched only in the given path.

If in the test we need to load a previously created file (.mscz, .musicxml, etc.), then by default the files will be searched in $USERDATA/AutobotTestingFiles. We can also set the environment variable MU_AUTOBOT_FILES_PATH, then the specified path will be searched.

Autobot can save various data during its work (files of created projects, reports, etc.). By default, data is saved to $USERDATA/AutobotData, to the corresponding subfolders.
We can also set the environment variable MU_AUTOBOT_DATA_PATH, then the specified path will be used to save the data.

See Resources.

Manual testing automation

For run scripts, go to menu Diagnostic->Autobot->Show scripts.... This will open a dialog with a scripts list (which were found in the given paths).

Click on the script to run it.

Each script usually performs some kind of test case, for example, create a Score with several instruments, add several notes, save and close the project. We can see how the script is executed and make decisions if everything is going to be done as expected or not.

Batch testing

The idea is to take a couple of thousand real user scores and perform a series of operations on each file (changing the presentation type, transposing, deleting measures, etc.). Make sure everything works as expected. And also creating a tool for controlling the change in drawing, that is, for each file, save the drawing data, modify the implementation, and compare the current drawing data with the previously saved ones (similar to vtests, but with more control and information about changes, with the possibility of filtering)

Writing scripts

To be continued