Tasking module

[mzient]

Category:

New feature (non-breaking change which adds functionality)

Description:

Tasking module

dali::tasking is a task execution engine which supports inter-task dependencies, semaphores and data passing.
Main concepts of the tasking API are:

• Task - a single-use object that encapsulates a function; it can depend on zero or more Waitable objects
• Waitable - an object for which a task can wait
• Scheduler - manages tasks and resolves their readiness state

Main components

A Task is also Waitable - that is, a task can depend on the completion of other task(s).
There are two kinds of Waitable objects - completion events and releasable objects.
A CompletionEvent (e.g. Task) is an object which remains acquirable after it's "completed". Performing Acquire operation on it does not alter the state of the object.
A Semaphore can be externally released (it's Releasable), but Acquire lowers the semaphore count, so the number of tasks that can acquire a semaphore before it's released is limited.

Acquisition of Waitable objects on behalf of waiting tasks is the duty of the Scheduler. By contrast, Release operation on Releasable objects can be performed in any context.

Data passing

Data can be passed between tasks. It's wrapped in a TaskResult type which stores either the value as std::any or the error as std::exception_ptr. An attempt to access the value with an exception present results in the exception being rethrown.
A Task wraps a function taking 0 or 1 argument (in the latter case it's this-like pointer to the Task) and returning a value of values.
Upon creation of the task, the caller defines the number of return values. There are two options:

• Scalar return value - in which case the return value can be any objects convertible to std::any. If the task returns std::any, it's stripped and the stored value is copied.
• Multiple return values - in this case the return value of the function must be either a collection or a tuple.
  A Task can Subscribe to the result of a producer task (which has to be done before the producer is submitted to the scheduler). The result can be obtained by calling task->GetInputValue<T>(index)

Obtaining results

When a task is added to the scheduler, a TaskFuture object can be obtained. Scheduler::AddSilentTask should be used if the result of the task is not needed. Calling TaskFuture::Value will wait for the task to complete and return the value produced by the task (or rethrow the exception).

Error handling

All errors thrown by the task functions are stored in the task results. An attempt to obtain results of a task that threw an exception will result in the exception being rethrown.

Additional information:

Usage with time dependencies only:

Executor ex;
ex.Start();
auto task1 = Task::Create([]() {
    cout << "Foo" << endl;
});
auto task2 = Task::Create([]() {
    cout << "Bar" << endl;
});
auto task3 = Task::Create([]() {
    cout << "Baz" << endl;
});
ex.AddSilentTask(task1);  // we're not interested in the result
ex.AddSilentTask(task2);
task3->Succeed(task1)->Succeed(task2);
auto future = ex.AddTask(task3);
future.Value<void>();

Usage with data dependencies

Executor ex;
ex.Start();
auto task1 = Task::Create([]() {
    return 12;
});
auto task2 = Task::Create([]() {
    return 30;
});
auto task3 = Task::Create([](Task *t) {
    return t->GetInputValue<int>(0) + t->GetInputValue<int>(1);
});
task3->Subscribe(task1)->Subscribe(task2);
ex.AddSilentTask(task1);
ex.AddSilentTask(task2);
auto future = ex.AddTask(task3);
cout << future.Value<int>() << endl;  // prints 42

Affected modules and functionalities:

All new code.

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-3934

[dali-automaton]

CI MESSAGE: [14339349]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14339349]: BUILD PASSED

[dali-automaton]

CI MESSAGE: [14358187]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14358187]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [14358187]: BUILD PASSED

[JanuszL]

Can you describe a convention in this list. What is the head, tail, next and prev.

[mzient]

I thought these were the most common names used in doubly-linked lists...
Saying that head_ denotes list's head doesn't improve anything - and the names in the comments can only go stale.

[JanuszL]

Sometimes head can point to itself if it is empty (we have an empty root that you cannot remove), or just be empty. That is what I mean.

[dali-automaton]

CI MESSAGE: [14452363]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14452363]: BUILD PASSED

[dali-automaton]

CI MESSAGE: [14476068]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14478063]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14482229]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14482229]: BUILD PASSED

[dali-automaton]

CI MESSAGE: [14538855]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14538855]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [14541090]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [14541090]: BUILD PASSED