integration testing

Integration Testing

Files placed in the directory Dedos/test/integration_tests are test suites that are run against entire DeDOS applications. They have the ability to stand up one or multiple runtimes, test within-runtime or inter-runtime communication, and inspect DeDOS internals, such as statistics or states.

Each file in test/integration_tests should test one application setup -- that is, it should be specific to one DFG file. Each application may have multiple setups on which it is tested, to ensure that the application works on one or more runtimes.

Furthermore, each application built for DeDOS should ideally have one or more test for each request pattern that it can serve.

Like Unit testing, integration tests rely on the Check framework. View Check documentation for lists of possible assertions within tests.

Test files

Each test should consist of three files:

• <test>.json: The DFG which is instantiated for the integration test should have an associated DFG file in JSON format
• <test>.mk: Lists the dependencies of the test (primarily the MSUs under test)
• <test>.c: The file with the integration tests themselves.

And optionally:

• <test>.yml: The associated yaml file from which the json file was created, for easier modification.

Dependency file (.mk)

The .mk file lists the files under test for this specific application. It should consist of the declaration of one variable, named <test>_DEPS, which lists the files which are manually included in the test file.

Note: The .mk file and associated _DEPS variable must share their prefix with the .c file containing the tests.

For instance, if the test file is my_dedos_app.c, which tests the msus msus/myapp/msu1.c and msus/myapp/msu2.c the dependency file should be called my_dedos_app.mk, and should contain the following definition:

my_dedos_app_DEPS:= msus/myapp/msu1.c
                    msus/myapp/msu2.c

Writing tests

File-under-test inclusion

At the top of the test file, include each of the source files under test (not just the header file, the actual .c file. This allows the test to have access to static variables).

e.g.

// Including files under test
#include "myapp/msu1.c"
#include "myapp/msu2.c"

// (Also include the integration test header)
#include "integration_test_utils.h"

Application DFG inclusion

At the bottom of the file, you must specify the json file which contains the DFG definition for the integration test.

This is done within the DEDOS_START_INTEGRATION_TEST_LIST(<dfg.json>) macro. It automatically assumes that the .json file specified is in the same folder as the test.

This macro will be followed with a list of the tests to perform, which is terminated with DEDOS_END_INTEGRATION_TEST_LIST().

e.g.

DEDOS_START_INTEGRATION_TEST_LIST("my_dedos_app.json")
/// List tests here
DEDOS_END_INTEGRATION_TEST_LIST()

Writing individual tests

Each test consists of two functions -- request and test. (This framework might be dubious and a bit unnecessary...)

The request function should perform whatever external action is necessary to set up the test, and the test function should ensure that DeDOS's internal state matches the expected values.

The request function should take no inputs, but can return any value. Request functions may be chained together to avoid repetition.

The test function should be declared with the macro START_DEDOS_INTEGRATION_TEST(<request_fn>) and end with END_DEDOS_INTEGRATION_TEST()

e.g.

int start_request() {
    // start connecting to webserver, for instance...
}
int finish_request() {
    int rtn = start_request();
    // finish connecting to webserver...
}

START_DEDOS_INTEGRATION_TEST(start_request) {
   // Make sure that MSU received the request
   // e.g.
   int n_seen = get_last_stat(MSU_ITEMS_PROCESSED, <msu_id>)
   ck_assert_int_eq(n_seen, 1);
}
START_DEDOS_INTEGRATION_TEST(finish_request) {
   int n_seen = get_last_stat(MSU_ITEMS_PROCSSED, <msu_id>)
   ck_assert_int_eq(n_seen, 2);
}

DEDOS_START_INTEGRATION_TEST_LIST("my_msu_app.json")
DEDOS_ADD_INTEGRATION_TEST_FN(start_request)
DEDOS_ADD_INTEGRATION_TEST_FN(finish_request)
DEDOS_END_INTEGRATION_TEST_LIST()

NOTE: This is an incomplete example. Variables such as the number of states in the MSU should also be tested.

Caveats!

Each integration test is run in its own fork! This is nice, in that all state is necessarily cleaned up after each test, and that you can rely on static global variables to pass information between request functions, if necessary.

Each runtime is run in its own fork, except for the runtime with id 1! This is necessary to ensure proper inter-runtime communication. However, this means that states are only accessible from runtime 1. For completeness, create a second integration test with the runtime IDs swapped to test the other runtime.

A helpful utility function is provided to aggregate the number of states recorded for all MSUs of a given type.

int type_num_states(struct msu_type *type);

will return the number of states across all instances of an MSU on runtime 1.