ATF Test Code

Table of Contents FunctionalTestBase Class AutomationSettings.xml File TestBase Class C Sharp Unit Tests Python Functional Tests Running a Python Script Topics in this section

ATF provides test code for its classes and samples. For automation, these tests use NUnit, which is a unit-testing framework for all .NET languages. For more information, see the NUnit site.

The Test folder in the ATF distribution contains two folders with different types of tests:

• FunctionalTests: Tests for ATF sample applications using Python scripts and NUnit. For information on these, see Python Functional Tests.
• UnitTests: Tests for ATF classes and capabilities using C# and NUnit. For details, see C Sharp Unit Tests.

Code for these tests resides in the Visual Studio solution Everything.sln in the ATF Test folder. This solution builds the test applications executables FunctionalTests.exe and UnitTests.exe.

These test applications run in a separate thread from the ATF application being tested.

ATF has several classes to facilitate writing applications that test or otherwise interact with ATF-based applications by using scripts. FunctionalTestBase and TestBase contain base methods that can be used in test applications. FunctionalTestBase is an abstract class that TestBase derives from, so you would use TestBase in your application. For descriptions of these classes' operation, see FunctionalTestBase Class and TestBase Class.

FunctionalTestBase Class

The abstract class Sce.Atf.Applications.FunctionalTestBase provides methods to set up and run Python scripts from C#. Note that a test application using FunctionalTestBase runs in a separate thread from the ATF application under test. FunctionalTestBase uses a proxy for the AutomationService component in the ATF application to actually execute scripts. To learn how this component provides script services, see AutomationService Component.

FunctionalTestBase's main methods are:

• SetupAppSettings(): To make tests more predictable (and hence reliable), automation needs to specify some settings. (For example, turn off the auto-load previously opened documents function, which can lead to popup dialogs if a previous document has an error or has moved). Do this by copying the AutomationSettings.xml file to the executable directory. When the application under test is launched, the settings service uses this file for application settings, bypassing any user-defined settings.
• LaunchTestApplication(): Launches the specified ATF application being tested in automation mode. "Automation mode" simply means passing an "-automation" flag to the executable, which tells the application to initialize the AutomationService component.
• Connect(): Connect to the AutomationService component, calling its Connect() method. It tries multiple times, to allow the ATF application time to launch and initialize. It obtains and returns an AutomationService proxy object that the test application can use.
• SetupScript(): One time set up before executing the actual script file.
• ExecuteStatementSafe(): Execute a Python statement on an application in a separate thread with the specified timeout. This method accomplishes this by calling the ExecuteStatement() method in the AutomationService proxy.
• ExecuteScriptSafe(): Execute a Python script on the application in a separate thread with the specified timeout. This method calls the ExecuteScript() method in the AutomationService proxy.
• ProcessScript(): Execute a given script by calling ExecuteScriptSafe() and verify that the result is "Success". The last message from the script needs to be "Success" to succeed.
• CloseApplication(): Close the ATF application being tested and check for errors in these steps:
  1. Discard all documents (close without saving).
  2. Check the ATF application for any error dialogs.
  3. Execute the StandardCommand.FileExit command, which closes the ATF application.
• ExecuteFullTest(): Set up the application for automation, run the script, process the result, and close the application. This method calls other methods to do the work: SetupAppSettings(), LaunchTestApplication(), Connect(), SetupScript(), ProcessScript(), and CloseApplication().
• TestCleanup(): Test cleanup, to be called after execution of each test, whether it passes or fails. The method disposes of the AutomationService and terminates the ATF application being tested if it is still running.

AutomationSettings.xml File

AutomationSettings.xml contains a set of default application settings to use for the application while it is being tested.

In its IInitializable.Initialize() method, the AutomationService component sets the SettingService.SettingsPath property to the path of AutomationSettings.xml. This results in the settings in AutomationSettings.xml being used for the application, rather than any user-prescribed settings.

Here is the standard AutomationSettings.xml provided with ATF:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<settings appName="" appVersion="3.0">
  <block id="Sce.Atf.Applications.AutoDocumentService">
    <value name="Auto New Document" type="Boolean">False</value>
    <value name="Auto-load Documents" type="Boolean">False</value>
  </block>
  <block id="9533d679-d825-4754-bc64-c468fe102873">
    <value name="Show Errors" type="Boolean">False</value>
    <value name="Enabled" type="Boolean">False</value>
  </block>
  <block id="18CABE31-4B86-4d10-9CD7-545616055E1B">
    <value name="Show File Association Dialog" type="Boolean">False</value>
  </block>
</settings>

These settings turn off automatically displaying the last document when the application starts (through the AutoDocumentService component), and suppress error messages and the file association dialog display.

For more information on these components, see SettingsService Component and AutomationService Component.

TestBase Class

The Sce.Atf.Applications.TestBase class derives from FunctionalTestBase. Its SetupScript() method overrides that of FunctionalTestBase to add some folders to the sys.path list of search paths through which the Python interpreter searches for modules. For instance, it adds the self-explanatory CommonTestScripts folder.

TestBase's TearDown() method indicates that the FunctionalTestBase.TestCleanup() method is called after each test. TearDown is an NUnit attribute used to mark a function that is performed after each test method is run:

[TearDown]
public virtual void TearDown()
{
    TestCleanup();
}

C Sharp Unit Tests

The unit test files are gathered in the UnitTests Visual Studio project. Its Program.cs file contains the Main() function that runs all the tests, displaying results in a Windows® Command Prompt. This project uses NUnit as the framework for running the tests.

Test files in the UnitTests folder use NUnit attributes to mark the tests to run. For example, here is the TestEmptyArray class in TestEmptyArray.cs:

[TestFixture]
public class TestEmptyArray
{
    [Test]
    public void TestInstance()
    {
        int[] test = EmptyArray<int>.Instance;
        Assert.NotNull(test);
        Assert.True(test.Length == 0);
    }
}

TestInstance() verifies that an instance of EmptyArray is indeed empty, raising assertions otherwise.

The TestFixtureAttribute class marks a class that contains tests. TestAttribute marks a method as a test method inside a class that has already been marked with TestFixtureAttribute. Each method marked with TestAttribute is automatically executed by NUnit.

Test files are in the UnitTests\Sce.Atf folder, and subfolders there test various areas, such as adaptation and the DOM. Some of the files test a particular class; some test a general capability. The C# methods in test files each run a particular test.

These test files and methods provide examples of C# tests for ATF. You can design similar tests for your application's classes and functionality.

Python Functional Tests

Python scripts in the FunctionalTests folder test ATF's sample applications, and you can use these techniques to test any application.

Similarly to UnitTests, the FunctionalTests project contains all the test files, both C# and Python. Its Program.cs file also contains the Main() function that runs all the tests and displays results in a Command Prompt window.

Running a Python Script

FunctionalTests has a set of C# files similar to those in UnitTests that have the test methods to be run with the [[Test] attribute. The difference is that each C# method runs a similarly named Python test script — instead of C# test code. All these files contain C# classes that derive from TestBase, and hence from FunctionalTestBase.

For example, the CircuitEditorTests class in CircuitEditorTests.cs has an AddAllItems() method that runs the Python script in AddAllItems.py:

[Test]
public void AddAllItems()
{
    ExecuteFullTest(ConstructScriptPath());
}

FunctionalTestBase.ConstructScriptPath() fashions a Python script path name from the calling method's name:

protected string ConstructScriptPath()
{
    string testName = new StackTrace().GetFrame(1).GetMethod().Name;
    string path = Path.Combine(GetScriptsDirectoryPath(), testName + ".py");
    return path;
}

The FunctionalTestBase.ExecuteFullTest() method described in FunctionalTestBase Class runs the Python script with the given path.

To learn more about script creation, see Writing Python Scripts for ATF Applications.

Topics in this section