SO 5.8 Experimental Testing

Introduction
An Experimental Support For Unit-testing Of Agents
- Design Goals
  - Independence From Testing Frameworks
  - Testing Scenarios As The Main Approach
A High-Level Description Of Testing Scenarios
- Scenario Steps
Step-by-Step Introduction By Examples
An Important Note On Completion Of A Step And The Whole Scenario

Introduction

SObjectizer-5 is in production use for several years. The experience shows that writing an agent is not a complex task. Writing the code of agent is, probably, the simplest part. But there is a more tricky part of work: testing the behavior of the agent.

Today the writing of unit-tests is the essential approach in software development. But how to write unit-tests for SObjectizer's agents?

It is a very hard question. And for a long time, there wasn't any help from SObjectizer in searching answer for it.

An Experimental Support For Unit-testing Of Agents

Experimental support for unit-testing of agents has been introduced in v.5.5.24. There are tools for writing "testing scenarios" and checking their successfulness. Those tools are intended to be used for writing unit-tests for user's agents.

Note that this support is an experimental feature even in SObjectizer-5.8. We think it is useful for testing agents. But, definitely, there can be some drawbacks in the design or implementation of these tools. Some important features can be missed.

We released this support to see how new features will be used "in the field", to collect some experience and feedback. We hope it allows us to make the support of unit-tests for agents more powerful and convenient.

So we ask our users to try this new functionality and share the experience with us.

Design Goals

Before we start the description of the new functionality it is necessary to point out some design goals and decisions we made. We hope it will help in the understanding of the usage principles of new features.

Independence From Testing Frameworks

There are several advanced and widely used unit-testing frameworks in C++ world like Boost.Test, gTest, Catch2, doctest and so on. We didn't want to bind our tools to one of them. We also didn't want to reinvent or recreate another one by ourselves.

Because of that, we have created tools for testing agents that way that these tools can be used with different unit-testing frameworks. In the examples below, we will use doctest framework but something similar can be written with Boost.Test or gTest or something else.

Testing Scenarios As The Main Approach

There were several open questions when we started our work on support for unit-testing. The main question was "How to test an agent?" Or even "What is a test for an agent?"

An agent is an autonomous entity. It works on some working context. We can only send a message to an agent. But send is an asynchronous operation. We can guess when the message arrives, but we don't know exactly when it happens.

Usually an agent work in cooperation with other agents. They can live on different contexts, they can send messages to each other. And we should have some way to see what happens when agent A receives a message from agent B.

It is hard to imagine how to write assertions in test cases with respect to all factors above. After some thinking, we ended up with an idea of "testing scenario". It was the only working approach we discovered. But if you have some ideas about alternatives please let us know.

A High-Level Description Of Testing Scenarios

A testing scenario contains several steps that should be activated and completed one after another. At the current version, the testing scenario is strictly sequential: step №2 can be activated only after the activation of step №1, step №3 only after activation of step №2 and so on. There can't be any branches/forks or cycles in the scenario.

A user should define and execute a testing scenario for a test case. There is no way to do some checking while the scenario is running. All necessary checks/assertions should be performed only after the finish of the scenario. So the typical usage of testing scenario in test cases looks like:

TEST_CASE("some_case") {
   /* creation of special testing SObjectizer Environment */
   /* creation of agents to be tested */
   /* definition of testing scenario */
   /* execution of testing scenario */
   /* checking the result of testing scenario execution */
}

Working time for a scenario should be limited. When this time is elapsed the scenario stops handling of any events and fixes the current state. This state should be checked to detect successfulness of scenario. If all steps were activated and completed then the scenario will be in completed state. It means that scenario succeed. If the final state of the scenario is not completed then scenario failed. Because of this typical test-case will look like:

TEST_CASE("some_case") {
   using namespace so_5::experimental::testing;

   testing_env_t env;
   ...
   env.environment().introduce_coop(...);
   ...
   env.scenario().define_step(...)...; // Definition of scenario's steps.
   ...
   // Execution of scenario.
   env.scenario().run_for(500ms);
   // Test scenario should complete successfully.
   REQUIRE(completed() == env.scenario().result());
   ... // Some other assertions.
}

Scenario Steps

Every scenario step has four important states: passive, preactivated, activated and completed.

All steps are in passive state at the beginning. When a step becomes the current step of the scenario it switches to preactivated state.

When some event happens inside SObjectizer (like handling a message by an agent) the information of this event is passed to the current (preactivated) step. The current step analyzes this information and can switch to activated state. When the current step becomes activated the scenario switches to the next step in the step chain (if there is such step). It means that activation of the current step leads to the changing of the current step: the next step in the chain becomes preactivated and all next events will be passed to it.

There is a slight difference between activation and completion of scenario steps. Activation means that the next step becomes the current step. For example, we can have steps S1, S2 and S3. Activation of S1 leads to preactivation of S2. All new events will go to S2. Activation of S2 leads to preactivation of S3 and all new events will go to S3.

Completion means that step completely finished its work after activation. A scenario can be completed successfully only when all steps were activated.

But why activation and completion are separate states of scenario step? Because sometimes completion can take some time and several steps can be activated while one step will be completed.

For example: suppose we have an agent Manager that reacts to message NewOrder. When Manager receives this message it should send a message LogNewOrder to agent Logger. So we can define two steps: the first one is the reaction of Manager to NewOrder and the second one is the reaction of Logger to LogNewOrder.

Suppose also that Manager should change its state from Free to Busy after handling NewOrder. We want to check this transition.

The problem is that Manager and Logger work on different contexts. It means that we can have the following sequence of actions:

Manager receives NewOrder (this activates the first step and precativates the second step).
Manager sends LogNewOrder.
Logger receives LogNewOrder (this activates the second step).
Manager switches its state.
Manager completes handling of NewOrder (this completes the first step).

It means that the first step requires more time for completion than the second step.

There are obviously steps that do not require additional time for completion. Like in the example above with Logger and LogNewOrder. Those simple steps switch from preactivated to completed state directly.

But more complex steps, like one with Manager agent, requires switching to activated state first. This allows preactivation of the next step. Then should be the switch from activated to completed state.

Step-by-Step Introduction By Examples

It seems that speaking about unit-testing of agents will be much easier with some simple examples at hands. Let's start with a very simple case and then go to more and more complex cases.

All the examples described below can be found in the following repository: so5_testing_demo.

Manager And Logger Example

We have already spoken about the example with Manager and Logger agents. Let's see how a simple unit-test for those agents can look like.

Source Code Of Agents

Just before we go to the code of unit-test let's take a quick look at the very simple implementation of our agents. There is the code of Logger agent:

class logger_t final : public so_5::agent_t {
public :
   struct log_new_order {
      std::string m_manager;
      std::string m_id;
   };

   using so_5::agent_t::agent_t;

   void so_define_agent() override {
      so_subscribe_self().event( [](mhood_t<log_new_order> cmd) {
         std::cout << "new order: manager=" << cmd->m_manager
            << ", id=" << cmd->m_id << std::endl;
      } );
   }
};

It simply receives log_new_order message and prints its content to the standard output.

And there is the code of Manager agent:

class manager_t final : public so_5::agent_t {
   const state_t st_free{ this, "free" };
   const state_t st_busy{ this, "busy" };

   const std::string m_name;
   const so_5::mbox_t m_logger;

public:
   struct new_order { 
      std::string m_id;
   };

   manager_t( context_t ctx, std::string name, so_5::mbox_t logger )
      :  so_5::agent_t{ std::move(ctx) }
      ,  m_name{ std::move(name) }
      ,  m_logger{ std::move(logger) }
   {}

   void so_define_agent() override {
      this >>= st_free;

      st_free.event( &manager_t::on_new_order );
   }

private:
   void on_new_order( mhood_t<new_order> cmd ) {
      this >>= st_busy;

      so_5::send< logger_t::log_new_order >( m_logger, m_name, cmd->m_id );
   }
};

This code is wordier but it is still very simple. Agent receives new_order message, switches from st_free to st_busy and sends log_new_order message to logger.

A Test Case For Manager/Logger

A test case for Manager and Logger agents can looks like as:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include <doctest/doctest.h>

#include <manager_logger/agents.hpp>

#include <so_5/experimental/testing.hpp>

namespace tests = so_5::experimental::testing;

using namespace std::chrono_literals;

TEST_CASE( "just-reacts_to" )
{
   tests::testing_env_t sobj;

   logger_t * logger{};
   manager_t * manager{};
   sobj.environment().introduce_coop(
      so_5::disp::active_obj::make_dispatcher(sobj.environment()).binder(),
      [&](so_5::coop_t & coop) {
         logger = coop.make_agent< logger_t >();
         manager = coop.make_agent< manager_t >("Fred", logger->so_direct_mbox());
      });

   sobj.scenario().define_step("order_received")
      .impact<manager_t::new_order>(*manager, "000-0001/0")
      .when(*manager & tests::reacts_to<manager_t::new_order>()
            & tests::store_state_name("manager"));

   sobj.scenario().define_step("order_logged")
      .when(*logger & tests::reacts_to<logger_t::log_new_order>());

   sobj.scenario().run_for(100ms);

   REQUIRE(tests::completed() == sobj.scenario().result());

   REQUIRE("busy" == sobj.scenario().stored_state_name("order_received", "manager"));
}

Let's see what is going on here.

The first two lines related to doctest framework. Then we include the definition of Manager and Logger agents. And then the most interesting parts begin.

First of all that is inclusion of testing-related part of SObjectizer:

#include <so_5/experimental/testing.hpp>

The standard include file so_5/all.hpp doesn't includes testing-related stuff. Because of that testing.hpp must be included manually.

All testing-related classes/functions/constants are defined in so_5::experimental::testing namespace. It is not easy to use such long namespace name in code and because of that we create much shorter synonym:

namespace tests = so_5::experimental::testing;

There is a test case that consists of the following sections:

The first section is a declaration of a special testing environment that is necessary to running agents in a controlling way:

   tests::testing_env_t sobj;

The class testing_env_t is very similar to wrapped_env_t -- it serves for the same purposes and has very similar API. The SObjectizer Environment is automatically started in the constructor of testing_env_t and automatically stopped in the destructor.

The second section is the creation of agents to be tested. There is nothing new for a SObjectizer's user:

   logger_t * logger{};
   manager_t * manager{};
   sobj.environment().introduce_coop(
      so_5::disp::active_obj::make_dispatcher(sobj.environment()).binder(),
      [&](so_5::coop_t & coop) {
         logger = coop.make_agent< logger_t >();
         manager = coop.make_agent< manager_t >("Fred", logger->so_direct_mbox());
      });

Every agent will be an active object -- it will work on its own worker thread.

Please note that we need to have pointers to agents to be tested. Because of that, we store those pointers to local variables.

The third section is a definition of the testing scenario:

   sobj.scenario().define_step("order_received")
      .impact<manager_t::new_order>(*manager, "000-0001/0")
      .when(*manager & tests::reacts_to<manager_t::new_order>()
            & tests::store_state_name("manager"));

   sobj.scenario().define_step("order_logged")
      .when(*logger & tests::reacts_to<logger_t::log_new_order>());

Every instance of testing_env_t already contains an instance of a testing scenario, there is no need to create scenario manually. This scenario instance is accessible via testing_env_t::scenario() method.

Two steps are defined for the testing scenario. The first step is a check for the reaction of Manager agent to incoming NewOrder message. To check this reaction we need to send an instance of NewOrder message. We do it by using impact method:

.impact<manager_t::new_order>(*manager, "000-0001/0")

This sentence tells the testing scenario that an instance of manager_t::new_order should be sent to the direct mbox of agent manager. All other arguments of impact are going to the constructor of new_order type.

It is a very good approximation to tell that impact can be seen as send that performed at the moment of preactivation of a step. Because of that impact has the same signature as send functions: it accepts a target of message/signal and parameters for the message's constructor. The target can be a reference to an agent, a reference to mbox or mchain.

Then we describe a trigger that activates our first step:

.when(*manager & tests::reacts_to<manager_t::new_order>()

We tell the scenario that the first step should be activated when Manager agent receives and handles NewOrder message. The message is expected from the direct mbox of the agent.

But there is additional part of that trigger:

& tests::store_state_name("manager"));

This part tells the scenario that when Manager agent finishes handling of NewOrder message the current state of the agent must be stored inside the scenario under the tag "manager".

Then we define the next step of the scenario:

sobj.scenario().define_step("order_logged")
   .when(*logger & tests::reacts_to<logger_t::log_new_order>());

This step is much simpler: we expect the only reaction to LogNewOrder that was sent to the direct mbox of Logger agent.

The fourth section of that test case is the execution of the testing scenario:

   sobj.scenario().run_for(100ms);

We allow our testing scenario to work no more than 100ms. For the modern hardware, it is a big time (even if our test is executed inside some virtual machine like VirtualBox or VMWare), so we can expect that 100ms is enough.

Method fun_for returns when scenario completes or when the specified time elapses.

The last, fifth section is a check for results of testing scenario execution:

   REQUIRE(tests::completed() == sobj.scenario().result());

   REQUIRE("busy" == sobj.scenario().stored_state_name("order_received", "manager"));

First of all, we are checking the successful completion of the scenario.

Then we check a stored state of the agent. This state was stored under the tag "manager" on step "order_received".

Since SO-5.8.3 there is a possibility to inspect message received by an agent. An example above can be extended this way:

TEST_CASE( "reacts_to-plus-inspect_msg" )
{
    tests::testing_env_t sobj;

    const std::string manager_name{ "Chris" };
    const std::string order_id{ "2024-10-11-0001" };

    logger_t * logger{};
    manager_t * manager{};
    sobj.environment().introduce_coop(
        so_5::disp::active_obj::make_dispatcher(sobj.environment()).binder(),
        [&](so_5::coop_t & coop) {
            logger = coop.make_agent< logger_t >();
            manager = coop.make_agent< manager_t >(manager_name, logger->so_direct_mbox());
        });

    sobj.scenario().define_step("order_received")
        .impact<manager_t::new_order>(*manager, order_id)
        .when(*manager
                & tests::reacts_to<manager_t::new_order>()
                & tests::store_state_name("manager"));

    sobj.scenario().define_step("order_logged")
        .when(*logger
                & tests::reacts_to<logger_t::log_new_order>()
                & tests::inspect_msg("log_new_order",
                    [&](const logger_t::log_new_order & msg) -> std::string {
                        if(msg.m_manager != manager_name)
                            return "manager_name mismatch, actual_value: " + msg.m_manager;
                        if(msg.m_id != order_id)
                            return "id mismatch, actual_value: " + msg.m_id;
                        return "OK";
                    })
        );

    sobj.scenario().run_for(100ms);

    REQUIRE(tests::completed() == sobj.scenario().result());

    REQUIRE("busy" == sobj.scenario().stored_state_name("order_received", "manager"));
    REQUIRE("OK" == sobj.scenario().stored_msg_inspection_result(
            "order_logged", "log_new_order"));
}

The step order_logged is extended: we add inspect_msg to reacts_to trigger. The inspect_msg specifies a lambda function to be called when the agent receives the message (but before an event handler of the agent is invoked). This lambda function has to return a std::string instance that will be stored inside the scenario for the step with specified tag name. In the case above it's "log_new_order" tag for "order_logger" step. This inspection result can then be obtained via stored_msg_inspection_result method (just like stored_state_name is used for store names of agent state).

A message inspector can return arbitrary value, the testing scenario just stores it. The proper interpretation of this value is up to the user.

Please note that inspect_msg can be used with mutable messages too. Something like:

sobj.scenario().define_step("my_step")
    .impact< so_5::mutable_msg<some_msg> >(*target_agent, ...)
    .when(*target_agent
            & tests::reacts_to< so_5::mutable_msg<some_msg> >()
            & tests::inspect_msg("msg_value", [](const some_msg & msg) -> std::string {...}));

Since SO-5.8.3 there is yet another iteresting possibility: if we want to inspect the value of a log_new_order message then we don't need an instance of the logger agent. We can just use a mbox:

TEST_CASE( "receives-plus-inspect_msg" )
{
    tests::testing_env_t sobj;

    const std::string manager_name{ "Chris" };
    const std::string order_id{ "2024-10-11-0001" };

    const auto logger_mbox = sobj.environment().create_mbox();

    manager_t * manager{};
    sobj.environment().introduce_coop(
        so_5::disp::active_obj::make_dispatcher(sobj.environment()).binder(),
        [&](so_5::coop_t & coop) {
            // No acual logger agent in the cooperation!
            manager = coop.make_agent< manager_t >(manager_name, logger_mbox);
        });

    sobj.scenario().define_step("order_received")
        .impact<manager_t::new_order>(*manager, order_id)
        .when(*manager
                & tests::reacts_to<manager_t::new_order>()
                & tests::store_state_name("manager"));

    sobj.scenario().define_step("log_new_order_sent")
        .when(logger_mbox
                & tests::receives<logger_t::log_new_order>()
                & tests::inspect_msg("log_new_order",
                    [&](const logger_t::log_new_order & msg) -> std::string {
                        if(msg.m_manager != manager_name)
                            return "manager_name mismatch, actual_value: " + msg.m_manager;
                        if(msg.m_id != order_id)
                            return "id mismatch, actual_value: " + msg.m_id;
                        return "OK";
                    })
        );

    sobj.scenario().run_for(100ms);

    REQUIRE(tests::completed() == sobj.scenario().result());

    REQUIRE("busy" == sobj.scenario().stored_state_name("order_received", "manager"));
    REQUIRE("OK" == sobj.scenario().stored_msg_inspection_result(
            "log_new_order_sent", "log_new_order"));
}

The receives trigger is available since SO-5.8.3. It can be applied to mboxes and it allows to check arrival of a message into a specified mbox. It also can be combined with inspect_msg modificator.

Please note that receives trigger can't be used with direct mboxes of agents.

Pinger And Ponger Example

The next example is again rather simple. There are two agents: Pinger and Ponger. Pinger sends a Ping request to Ponger, and Ponger replies with Pong answer. Basically the logic somewhat similar to Manager/Logger example. But there is an important difference: in Manager/Logger example the first message was sent by us, and in Piner/Ponger example the first Ping request is sent automatically by Pinger agent in so_evt_start method.

The Source Code Of Agents

The Pinger/Ponger agents and corresponding request/reply signals look like:

struct ping final : so_5::signal_t {};
struct pong final : so_5::signal_t {};

class pinger_t final : public so_5::agent_t {
   so_5::mbox_t m_target;
public :
   pinger_t( context_t ctx ) : so_5::agent_t{ std::move(ctx) } {
      so_subscribe_self().event( [this](mhood_t<pong>) {
         so_deregister_agent_coop_normally();
      } );
   }

   void set_target( const so_5::mbox_t & to ) { m_target = to; }

   void so_evt_start() override {
      so_5::send< ping >( m_target );
   }
};

class ponger_t final : public so_5::agent_t {
   so_5::mbox_t m_target;
public :
   ponger_t( context_t ctx ) : so_5::agent_t{ std::move(ctx) } {
      so_subscribe_self().event( [this](mhood_t<ping>) {
         so_5::send< pong >( m_target );
      } );
   }

   void set_target( const so_5::mbox_t & to ) { m_target = to; }
};

The only thing that could be explained here is the presence of set_target methods.

We will use the direct mboxes of Pinger/Ponger agents. So we have to tell each agent the direct mbox of the another. We can't do that in the constructors because it is necessary to create an agent first and only then the direct mbox can be obtained. We create both agents and then bind them one to another by using set_target method.

A Test Case For Pinger/Ponger

A test case for checking the behavior of Pinger/Ponger agents can look like:

TEST_CASE( "ping_pong" )
{
   tests::testing_env_t sobj;

   pinger_t * pinger{};
   ponger_t * ponger{};
   sobj.environment().introduce_coop([&](so_5::coop_t & coop) {
      pinger = coop.make_agent< pinger_t >();
      ponger = coop.make_agent< ponger_t >();

      pinger->set_target( ponger->so_direct_mbox() );
      ponger->set_target( pinger->so_direct_mbox() );
   });

   sobj.scenario().define_step("ping")
      .when(*ponger & tests::reacts_to<ping>());

   sobj.scenario().define_step("pong")
      .when(*pinger & tests::reacts_to<pong>());

   sobj.scenario().run_for(100ms);

   REQUIRE(tests::completed() == sobj.scenario().result());
}

There we again can see five sections described above: the creation of the testing environment, creation of agents (with binding them one to one), the definition of testing scenario, the execution of the scenario and the checking of scenario success.

But there is one thing that is hidden but very important for the understanding of this test case.

As we see from the source code of Pinger agent, the Pinger sends Ping request in so_evt_start. It means that in normal circumstances Pinger and Ponger can complete message exchange even before the return from introduce_coop. But it does not happen here.

Why?

When testing_env_t is used all agents are frozen after the registration if the testing scenario is not started yet. This means that agents are present in the SObjectizer Environment but they can't handle any events (even so_evt_start is not called). It is possible to send a message to a frozen agent, but this message will wait in some event queue while the agent will be unfrozen.

All agents those are registered before a call to run_for will be automatically unfrozen when run_for is called. It means that so_evt_start for Pinger agent will be called only when we call run_for in our scenario.

So the key point of that simple example is: before the call to run_for all agents are frozen, they are present but can't handle messages (even so_evt_start is not called). Agents will be unfrozen when run_for is called.

Note. The unfreeze of an agent is also performed when testing_env_t has to be destroyed. For example in that case:

TEST_CASE( "incomplete test case" )
{
   tests::testing_env_t sobj;

   pinger_t * pinger{};
   ponger_t * ponger{};
   sobj.environment().introduce_coop([&](so_5::coop_t & coop) {
      pinger = coop.make_agent< pinger_t >();
      ponger = coop.make_agent< ponger_t >();

      pinger->set_target( ponger->so_direct_mbox() );
      ponger->set_target( pinger->so_direct_mbox() );
   });
//FIXME: write an actual code of test case.
   return;
}

Pinger/Ponger agents will be unfrozen on return statement.

Dining Philosophers Example

Another example is a classical concurrent programming problem: Dining Philosophers.

One solution on that problem implemented by using SObjectizer agents can be found here. There are two types of agents: Fork for the representation a resource and Philosopher for the representation of a resource consumer. We will not show the source code of those agent types here. Agents' logic will be briefly described in the corresponding sections below.

A Test Case For Fork Agent

The Fork agent has two states and reacts to two types of incoming messages. When an agent in Free state it reacts to a Take message, switches to Taken state and responds by a Taken signal. Signal Put is just ignored in Free state.

When an agent in Taken state it reacts to Take message differently: it stays in Take state and responds with a Busy signal. Signal Put is handled in Taken state: agent switches to Free state.

It is important to say that message Take contains a mbox inside. This mbox is used for Taken and Busy replies.

Let's try to check the behavior of Fork agent with that test case:

TEST_CASE( "fork" )
{
   class pseudo_philosopher_t final : public so_5::agent_t {
   public:
      pseudo_philosopher_t(context_t ctx) : so_5::agent_t{std::move(ctx)} {
         so_subscribe_self()
            .event([](mhood_t<msg_taken>) {})
            .event([](mhood_t<msg_busy>) {});
      }
   };

   tests::testing_env_t sobj;

   so_5::agent_t * fork{};
   so_5::agent_t * philosopher{};
   sobj.environment().introduce_coop([&](so_5::coop_t & coop) {
      fork = coop.make_agent<fork_t>();
      philosopher = coop.make_agent<pseudo_philosopher_t>();
   });

   sobj.scenario().define_step("put_when_free")
      .impact<msg_put>(*fork)
      .when(*fork & tests::ignores<msg_put>());

   sobj.scenario().define_step("take_when_free")
      .impact<msg_take>(*fork, philosopher->so_direct_mbox())
      .when_all(
         *fork & tests::reacts_to<msg_take>() & tests::store_state_name("fork"),
         *philosopher & tests::reacts_to<msg_taken>());

   sobj.scenario().define_step("take_when_taken")
      .impact<msg_take>(*fork, philosopher->so_direct_mbox())
      .when_all(
         *fork & tests::reacts_to<msg_take>(),
         *philosopher & tests::reacts_to<msg_busy>());

   sobj.scenario().define_step("put_when_taken")
      .impact<msg_put>(*fork)
      .when(
         *fork & tests::reacts_to<msg_put>() & tests::store_state_name("fork"));

   sobj.scenario().run_for(100ms);

   REQUIRE(tests::completed() == sobj.scenario().result());

   REQUIRE("taken" == sobj.scenario().stored_state_name("take_when_free", "fork"));
   REQUIRE("free" == sobj.scenario().stored_state_name("put_when_taken", "fork"));

}

This test case somewhat similar to the test cases described above, but there are some important differences.

First of all Fork agent is intended to be used with Philosopher agent but we can't use an actual Philosopher in this test case because Philosopher will do its own logic and this will disturb us. So instead of actual Philosopher agent we will use an imitation: agent that accepts messages, but does nothing (so we use mock-object instead of actual Philosopher):

class pseudo_philosopher_t final : public so_5::agent_t {
public:
   pseudo_philosopher_t(context_t ctx) : so_5::agent_t{std::move(ctx)} {
      so_subscribe_self()
         .event([](mhood_t<msg_taken>) {})
         .event([](mhood_t<msg_busy>) {});
   }
};

We will use the direct mbox of that agent in messages sent to Fork agent. We also will use this agent in the scenario to check replies from Fork agent.

The next difference is the presence of ignores trigger:

.when(*fork & tests::ignores<msg_put>());

This triggers will be activated when a message of type msg_put will be sent to Fork agent and will be ignored (rejected) by the agent. It can be seen as opposite to reacts_to trigger.

Please note that since SO-5.8.3 the ignores trigger can be used with inspect_msg modificator.

And yet another difference with test cases described above is when_all used to trigger some scenario steps:

sobj.scenario().define_step("take_when_free")
   ...
   .when_all(
      *fork & tests::reacts_to<msg_take>() & tests::store_state_name("fork"),
      *philosopher & tests::reacts_to<msg_taken>());

Method when_all tells that a step will be activated only when all triggers are activated. In that particular case for activation of step with name take_when_free it is necessary that:

Fork agent handles message msg_take from its direct mbox;
message msg_taken is sent as the reply to "Philosopher" agent.

Now we can speak about the test case itself. Fork agent has simple logic: when it is in Free state it doesn't react to Put message. So we check this behavior by put_when_free step:

sobj.scenario().define_step("put_when_free")
   .impact<msg_put>(*fork)
   .when(*fork & tests::ignores<msg_put>());

Fork agent should react to Take message when it is Free state. When it receives Take message it should change the state and should answer by Taken message. This is checked by take_when_free step:

sobj.scenario().define_step("take_when_free")
   .impact<msg_take>(*fork, philosopher->so_direct_mbox())
   .when_all(
      *fork & tests::reacts_to<msg_take>() & tests::store_state_name("fork"),
      *philosopher & tests::reacts_to<msg_taken>());

Then, when Fork agent in Taken state it should handle repeated Take messages different way: it should respond with Busy message. We check this by take_when_taken step:

sobj.scenario().define_step("take_when_taken")
   .impact<msg_take>(*fork, philosopher->so_direct_mbox())
   .when_all(
      *fork & tests::reacts_to<msg_take>(),
      *philosopher & tests::reacts_to<msg_busy>());

And the last step is checking of reaction to Put message in Taken state:

sobj.scenario().define_step("put_when_taken")
   .impact<msg_put>(*fork)
   .when(
      *fork & tests::reacts_to<msg_put>() & tests::store_state_name("fork"));

To complete the test case we check names of states stored at take_when_free and put_when_taken steps because Fork agent is expected to change its states on those steps.

A Test Case For Philosopher Agent

The last test case we discuss is a test case for Philosopher agent. This agent behaves as follows:

thinks for some time;
tries to take the left fork;
if the left fork is taken it tries to take the right fork;
if the right fork is taken it eats for some time;
then it releases both forks and starts thinking for some time.

There are some points where something can go wrong: for example, the left fork may be taken by another Philosopher. In that case, Philosopher should return to thinking. But in that particular test case, we will check only successful scenario.

The whole test case will look like:

TEST_CASE( "philosopher (takes both forks)" )
{
   tests::testing_env_t sobj{
      [](so_5::environment_params_t & params) {
         params.message_delivery_tracer(
               so_5::msg_tracing::std_cout_tracer());
      }
   };

   so_5::agent_t * philosopher{};
   so_5::agent_t * left_fork{};
   so_5::agent_t * right_fork{};

   sobj.environment().introduce_coop([&](so_5::coop_t & coop) {
      left_fork = coop.make_agent<fork_t>();
      right_fork = coop.make_agent<fork_t>();
      philosopher = coop.make_agent<philosopher_t>(
         "philosopher",
         left_fork->so_direct_mbox(),
         right_fork->so_direct_mbox());
   });

   auto scenario = sobj.scenario();

   scenario.define_step("stop_thinking")
      .when( *philosopher
            & tests::reacts_to<philosopher_t::msg_stop_thinking>()
            & tests::store_state_name("philosopher") )
      .constraints( tests::not_before(250ms) );

   scenario.define_step("take_left")
      .when( *left_fork & tests::reacts_to<msg_take>() );

   scenario.define_step("left_taken")
      .when( *philosopher
            & tests::reacts_to<msg_taken>()
            & tests::store_state_name("philosopher") );

   scenario.define_step("take_right")
      .when( *right_fork & tests::reacts_to<msg_take>() );

   scenario.define_step("right_taken")
      .when( *philosopher
            & tests::reacts_to<msg_taken>()
            & tests::store_state_name("philosopher") );

   scenario.define_step("stop_eating")
      .when( *philosopher
            & tests::reacts_to<philosopher_t::msg_stop_eating>()
            & tests::store_state_name("philosopher") )
      .constraints( tests::not_before(250ms) );

   scenario.define_step("return_forks")
      .when_all( 
            *left_fork & tests::reacts_to<msg_put>(),
            *right_fork & tests::reacts_to<msg_put>() );

   scenario.run_for(1s);

   REQUIRE(tests::completed() == scenario.result());

   REQUIRE("wait_left" == scenario.stored_state_name("stop_thinking", "philosopher"));
   REQUIRE("wait_right" == scenario.stored_state_name("left_taken", "philosopher"));
   REQUIRE("eating" == scenario.stored_state_name("right_taken", "philosopher"));
   REQUIRE("thinking" == scenario.stored_state_name("stop_eating", "philosopher"));
}

There is plenty of code but there are only a few new things we haven't discussed yet. The testing scenario contains many steps but every step is trivial and has a simple trigger(s). Maybe only two things in this test case should be mentioned separately.

The first one is usage of another constructor of testing_env_t class:

tests::testing_env_t sobj{
   [](so_5::environment_params_t & params) {
      params.message_delivery_tracer(
            so_5::msg_tracing::std_cout_tracer());
   }
};

This constructor allows to tune parameters for SObjectizer Environment. In that case, we turn message delivery on with logging via std::cout. Message delivery tracing can be very useful in complex scenarios of message interchange. So if we want to see what happens during message exchange we can turn message delivery tracing on and then analyze the trace.

The second thing we have to mention is usage of constraints method:

scenario.define_step("stop_thinking")
   .when( *philosopher
         & tests::reacts_to<philosopher_t::msg_stop_thinking>()
         & tests::store_state_name("philosopher") )
   .constraints( tests::not_before(250ms) );

This method allows setting some constraints for a step. The step can be activated only when all specified constraints are fulfilled.

There are two constraints that can be used: not_before and not_after. They can also be used together:

scenario.define_step(...)
   .constraints(tests::not_before(250ms), tests::not_after(1250ms))
   ...;

but SObjectizer doesn't check the correctness of not_before and not_after pair. It means that one can write not_before(1s) with not_after(250ms) and this won't be detected as an error by SObjectizer.

When we define a step with some constraints, like this one:

scenario.define_step("stop_thinking")
   .when( *philosopher
         & tests::reacts_to<philosopher_t::msg_stop_thinking>()
         & tests::store_state_name("philosopher") )
   .constraints( tests::not_before(std::chrono::milliseconds(250)) );

then SObjectizer will check constraints first and only then if constraints are fulfilled, step's triggers will be checked. For example, if message StopThinking arrives before 250ms the trigger for step stop_thinking won't be activated.

Time for not_before and not_after is counted from the moment of step preactivation. So if we send a couple of delayed messages:

so_5::send_delayed<some_message>(first_agent, 150ms, ...);
so_5::send_delayed<some_message>(second_agent, 250ms, ...);

and want to use them as triggers for sequential steps with constraints not_before then we have to write like that:

scenario.define_step("first")
   .constraints( tests::not_before(150ms) )
   .when( first_agent & tests::reacts_to<some_message>() );
scenario.define_step("second")
   .constraints( tests::not_before(100ms) )
   .when( second_agent & tests::reacts_to<some_message>() );

Please note that additional care should be taken when timeouts are used. For example, if we have something like that:

// Define scenario.
scenario.define_step("first")
   .constraints( tests::not_before(150ms) ) // Point (3).
   .when( first_agent & tests::reacts_to<some_message>() );
scenario.define_step("second")
   .constraints( tests::not_before(100ms) )
   .when( second_agent & tests::reacts_to<some_message>() );
// Point (1). Initiate some actions those should impact our scenario.
so_5::send_delayed<some_message>(first_agent, 150ms, ...);
so_5::send_delayed<some_message>(second_agent, 250ms, ...);
// Point (2). Run scenario.
scenario.run_for(1s);
// Check the result.
REQUIRE(tests::completed() == scenario.result());

we can have cases when our scenario will fail. It is because on some machines (virtual machines for example) several milliseconds can be spent between points (1) and (2). It means that when the first instance of some_message will be delivered to first_agent the constraint at point (3) won't be fulfilled: there can be less than 150ms between preactivation of the first step and arrival of some_message instance.

An Important Note About ignores() Trigger

Usage of ignores trigger requires more attention than the usage of reacts_to trigger. Because ignores activates only when a message is delivered to an agent but rejected by this agent. For example, an agent has two states and has a subscription to that message type only in one state.

But there are cases when a message won't be delivered to an agent at all. For example, if an agent has no subscription to message yet. Or the message was rejected by a delivery filter. Or the message was rejected by overload protection mechanism (limit_then_drop for example).

In such cases ignores trigger won't be activated because there won't be a delivery of a message to the receiver.

An Important Note On Completion Of A Step And The Whole Scenario

As mentioned earlier the scenario completes when all steps are activated.

It means that the scenario doesn't wait for the completion of steps by default. So agents inside the testing environment can still work upon return from run_for.

Usually it isn't a problem. But sometimes it can lead to mistakes. Usually it happens when we're trying to share some resources with a working agent. Something like:

struct ack_for_value {
    std::string * m_receiver;
    ...
};

// Agent that reacts to ack_for_value.
class my_agent final : public so_5::agent_t {
    ...
    void on_ask_for_value(const ask_for_value & msg) {
        msg.m_receiver->assign("new-value");
    }
};
...
// Somewhere in a test case.
my_agent * target = ...
std::string value_receiver;

sobj.scenario().define_step("ask_value")
    .impact<ask_for_value>(*target, &value_receiver)
    .when(*target & tests::reacts_to<ask_for_value());
// No other steps.
sobj.scenario().run_for(100ms);

REQUIRE(tests::completed() == sobj.scenario().result());

// OOPS!
REQUIRE("new-value" == value_receiver);

The problem is that my_agent::on_ask_for_value may still work while we attempt to access value_receiver content. It's because the "ask_value" step activates when the agent receives a message but before the event-handler of the agent is called. It means that the scenario may complete its work before invocation of the agent's event-handler.

If we want to wait completion of the agent event-handler before the completion of the scenario then we have to use wait_event_handler_completion modificator introduced in SO-5.8.3:

sobj.scenario().define_step("ask_value")
    .impact<ask_for_value>(*target, &value_receiver)
    .when(*target & tests::reacts_to<ask_for_value()
        & tests::wait_event_handler_completion());

Provide feedback

Saved searches

Use saved searches to filter your results more quickly