Watchdogs

[edwardalee]

Watchdogs

A watchdog is specified as follows:

    watchdog watchdog_name(min_timeout) {= optional code body =}

The min_timeout parameter is optional and defaults to zero. In the target language, there are two API functions:

    reaction(... triggers ...) -> watchdog_name {=
        lf_watchdog_start(watchdog_name, additional_timeout);
        lf_watchdog_stop(watchdog_name);
    =}

Let timeout = additional_timeout + min_timeout. When you call lf_watchdog_start, a physical timer gets set to expire at physical time equal to the current logical time t plus the timeout. If lf_watchdog_stop is called before the physical timer expires, then the timer is canceled. If lf_watchdog_start is called again before the physical timer expires, then it simply gets reset to expire at the newly specified time.

If and when the physical timer expires, two things happen. First, if an optional code body is specified with the watchdog declaration, then that code body is executed. This code will be mutually exclusive with any reactions in the same reactor. It has access to the self struct, and can therefore read parameters and read and write state variables. It does not have access to ports, actions, or watchdogs.

Second, any reactions that list watchdog_name among their triggers will be invoked at the current logical time plus one microstep. NOTE: An alternative would be schedule it like a physical action. Which is better?

Usage patterns:

Lag monitor:

The goal is to ensure that logical time does not lag too much behind physical time.

reactor LagMonitor {
    watchdog w(100 ms) {=
        panic();
    =}
    timer t(0, 10 ms);
    reaction(t) -> w {=
        lf_watchdog_start(w, 0);
    =}
}

The panic() function will be invoked if an only if the reaction to timer t does not get invoked by physical time n x 10 + 100 ms, where n is 0 for the first timer tick, 1 for the second, etc. Note that this is distinct from a deadline, where the handler is invoked if the n-th reaction to t is invoked after physical time n x 10 plus the deadline time.

One-Time Lag Monitor:

The above example restarts the watchdog on every timer tick, but we may not want to restart the watchdog after it expires once. This is accomplished by:

reactor OneTimeLagMonitor {
    state expired:bool(false);
    watchdog w(100 ms) {=
        panic();
        self->expired = true;
    =}
    timer t(0, 10 ms);
    reaction(t) -> w {=
        if (!self->expired) {
            lf_watchdog_start(w, 0);
        }
    =}
}

Notice that there is a race condition between the two code bodies. If the watchdog timer expires right around the time that the timer reaction is being invoked, then the order in which the watchdog body and the reaction body are invoked is not defined. If the timer reaction is invoked first, then the invocation of the watchdog body will be canceled.

Mode Switch

FIXME: Mode switch example with a reaction triggered by the watchdog.

Implementation

The watchdog has a state variable expiration that is a time initialized to NEVER.
Sketch:

void lf_watchdog_start(Watchdog* w, interval_t additional_timeout) {
    acquire mutex;
    w->expiration = lf_time_logical() + w->min_expiration + additional_timeout;
    if (thread is not running) {
        create one and start it.
    }
    release mutex;
}

The thread body looks like this:

    acquire_mutex;
    while(lf_time_physical() < w->expiration) {
        T = calculate physical time until expiration;
        release mutex;
        lf_nanosleep(T);
        acquire mutex;
   }
   invoke watchdog body.
   release mutex;

Questions and Discussion

Note that the watchdog body has no access to actions and hence cannot call schedule. This is why the watchdog is also a trigger.

Note that the watchdog trigger will occur at the next available logical tag, as if a logical action had been asynchronously scheduled (something that is not allowed). The reason for this is that if we were instead to treat it as a physical action, it could end up on the event queue behind many other events that will have to be handled before you can react to the watchdog expirations.

Note that it might be tempting to use this mechanism to achieve anytime computation, where you set a physical time bound on the execution of a reaction and then interrupt it when that time expires. However, this will not work because the watchdog handler is mutually exclusive with the reactions in the same reactor. To achieve anytime computation, use FIXME (checking deadlines during runtime).

In case we want a mechanism that aborts a rogue long running reaction, a general method might be to have the runtime call setjmp prior to executing a reaction. Then exception handling code could call longjmp to abort the reaction. Problems:

1. This still cannot be done in a watchdog handler in the same reactor because of the mutex with the reaction.
2. How to get a handle on the worker thread executing the reaction?
3. Memory leaks could result if the reaction has allocated memory.

How to handle shutdown? Stop all watchdogs?

[erlingrj]

The implementation looks good to me. We will also require to acquire the mutex before invoking any other reaction in the same Reactor.

For the single-threaded targets we would need to go to sleep on HW timer and have the watchdog body be invoked either in:

1. If not executing any Reaction in Reactor where Watchdog is instantied: Execute Watchdog body in Timer ISR
2. If reaction is executing. Set a flag in the self_struct. After finishing any Reaction invokation, check watchdog flags and possibly invoke watchdog body

I think this means that we need a unique timer (or timer channel) for each watchdog. This is how I would do it:

1. We would need to initialize the watchdog timers at boot. This would require a platform API for acquiring and setting up HW timers. This is not possible with only the systick timer and implies that we should build platform support ontop of MCU SDKs.

static int watchdog_to_timer_id[N_WATCHDOGS];
static Watchdog* timer_id_to_watchdog[N_HW_TIMERS];

lf_watchdog_init(Watchdog *w) {
  int timer_id = lf_init_hw_timer();
  watchdog_to_timer_id[w->id] = timer_id;
  timer_id_to_watchdog[timer_id] = w;
}

2. Starting the watchdog is just scheduling a future interrupt on one of the HW timers. We have to take care of overflow etc.

void lf_watchdog_start(Watchdog* w, interval_t additional_timeout) {
  lf_hw_timer_set_timeout(watchdog_to_timer_id[w->id], w->min_timeout + additional_timeout);
  // By setting pending to false we now implement Peters idea about handling concurrent timeout and reset
  w->pending = false;
}

3. The ISR could look something like this

void lf_hw_timer_isr(int timer_id) {
  Watchdog *w = timer_id_to_watchdog[timer_id];
  // Check if busy with reaction
  if (w->parent->executing_reaction == NULL) {
    (w->body)();
  } else {
    w->pending = true;
  }
}

4. We would have to check that flag whenever we finish executing a reaction

void lf_invoke_reaction(Reaction *r) {
  self->executing_reaction = r;
  (r->body)();
  self->executing_reaction = NULL;
  for w in self->watchdogs {
    if (w->pending) {
     (w->body)();
     w->pending = false;
   }
}

I think there are many edge cases here and the design for single-threaded runtime must be thought through more thoroughly. But an important observation is that this is not feasible without using Timer peripherals external to the CPU which mean that we should probably re-evaluate the idea of build bare-metal support just on the CPU rather than on the SDK

[erlingrj]

Steps

Adding new syntax

• Update grammar with Watchdog spec
• Code-generation steps

Reactor-c runtime

• Add a mutex to each reactor
• code-generate initialization of the mutex for all reactors
• acquire mutex (if present) before reaction invocation. And release after finished.
• Code-generate mutex init only for reactors with Watchdogs, also only acquire mutex if there is a watchdog (if watchdog != NULL)
• Implement lf_watchdog_start and lf_watchdog_stop