Authority on information about evaluations/experiments

[sondreso]

We need a higher level concept than the ensemble evaluator to control what evaluations are running, and provide information about this to other modules and clients.

In particular, we are running into issues because we are unable to differentiate between the evaluation not having started yet and the evaluation already being done.

• Phases should be events
• Be stoppable. Today only the queue is allowed to be killed, all parts of the experiments should be cancelable.
• Support more modes of re-runs/restarts. See Restart previously failed realizations #358
• In the same vein as above, implement support for restarting failed runs from the last successful forward model componen as described in Restart failed runs from last successful forward model component #609
• Handle long running jobs

[markusdregi]

This is a later in the process, but still kind of related to #1340

[oysteoh]

Dealing with sockets and ports, i have a "finding" i can add to this issue.

Due to the evaluator using an instance of the config with host / port to acquire a socket we need to bind with reuse_addr flag. This because the binding could happen multiple times towards the same port. Without the flag we could experience flakyness due to the fact we never know how long it will take the server to shut down the socket and make it available for binding again. With reuse_addr the socket can be bound to even if it is in TIME_WAIT state.

[oyvindeide]

Will mark this as closed to BaseRunModel (experiment) is now responsible for forwarding events from both EE and the updates.