Review status related fields in H and calc_status

[shuds13]

This issue now relates to the wiki

We currently have several fields related to the current status of an evaluation such as the protected fields 'given' and 'returned' and also some allocation function specific fields such as 'given_back'.

We also have a calc_status optionally returned by a sim_f or gen_f.

Also, the branch cwp/async_build adds H fields for 'cancel' and 'kill_sent' as protected fields (they are used by the manager to send kill signals to any worker with an active cancelled evaluation).

We should consider re-organizing into a better set of options. This could potentially change fields and so could be an API breaking change.

My proposal is to express for each H row, the fields below, where field categories are separated for mutual exclusivity within category.

1 Schedule status: (currently in sep. fields 'given', 'returned', 'allocated', 'given_back; ....
Merge to one field - name to be determined - could include..

• 'Null'
• 'generated'
• 'allocated' [*edit - this is alloc only currently, and probably not needed - could be dealt with locally in alloc_f]
• 'given_to_sim'
• 'returned_from_sim'
• 'given_back_to_gen' [*edit - this is alloc only currently, perhaps should be equiv. to given_to_sim?]

*edit - allocated and given_back are used in alloc functions to prevent points being processed more than once. I think that if we can assume allocated points are always sent out, then allocated could be replaced by a local variable in alloc_f. Likewise if we make given_back (or given_back_to_gen) equivalent to given which is set in core libE (libE_manager), then that could be treated in a similar way (combined with a local modifier in the alloc_f).

I am trying to propose names that avoid ambiguity (eg. given to what?). Options could be a string or named integer. A problem is whether we want to keep flexibility for user to set their own. Which could be added fields in addition to these.

2 calc_status -> sim_status? - move to an H field
calc_status is currently linked to a run of sim_f/gen_f - which can include multiple H rows - but we may want a field in H that would obviously relate to one sim_id.

• 'Not set'
• 'Finished' (success)
• 'Failed'
• 'Killed' (timeout, on user condition, by manager/gen???)

• Note we would need to separate out what is currently returned to gen, like the FINISHED_PERSISTENT_GEN_TAG is given as a calc_status and printed to libE_stats, but also used as an message to manager. These things would need to be decoupled, and decide what is printed to libE_stats.
• Timing in libE_stats relates to a call to sim_f or gen_f (which can potentially involve multiple H rows), and prints a status (if it is set). How would we reconcile this. Note that individual tasks run by the executor also get timed, but we do not automatically have a timing uniquely related to one H row.

3 Cancel
True of False.
Due to asynchronicity - 'cancelled' could be in various combinations of states in 1 & 2.
Should this have the possibility to have different options (e.g. cancel_with_kill, cancel_no_kill)

4 "kill_sent" (or could it be in 3).
see https://github.com/Libensemble/libensemble/blob/cwp/async_build/libensemble/libE_manager.py
Need to mark when a kill signal has been sent to a running calculation so do not send more than one. This is also complicated by potentially many-to-one mapping of H row to a sim_f. Currently the code sends a kill to a sim_f if it contains a cancelled row, Does not select sim_ids to kill within one sim_f call.

*note in alloc_f the variable "task_avail" is used quite often. May want to change 'task' to 'sim'.

[jmlarson1]

I like this. Some thoughts: I would not have 'allocated'because the manager sends them to the workers as soon as they are requested.

Consider having 'cancelled' being 'cancel_requested' or rather 'kill_requested' to match 'killed' and 'kill_sent'

Otherwise, it looks great.

[shuds13]

Consider having 'cancelled' being 'cancel_requested' or rather 'kill_requested' to match 'killed' and 'kill_sent'

I was thinking that kill_requested is not quite right, in that if a task is cancelled and it has not yet been sent to a sim worker, then it does not get sent (as long as supported by the allocation function). So it only 'kills' a running task if its already sent out.

The field as I currently have it is cancel. My intuition of cancel is like I have cancelled my order. It might be that the order has not been processed, and on receiving the cancel they will not processes it. It might be getting processed and they abort (kill), or it might already be on the way to being delivered to my house, and I'll get it anyway.

In a literal sense, I suppose cancel implies successfully cancelled. Maybe cancel_requested or request_cancel is better to signify intent, but is the extra length worth it? Anyone else got a view on this?

@jlnav @wildsm

Bear in mind, the allocation function could decide whether to return results that got processed, despite cancel being set, back to the generator, and if so the gen (who decided the cancel) could decide whether to include these (despite cancel being True on that row), or could still discard these results.

[jlnav]

I like cancel_requested, it describes the intent of cancelling, but it's not always surprising if a point labeled as such returns.

[shuds13]

I am wondering if 'restart' should be added to the scheduling options? I'm thinking that if a point was cancelled, but then was set to uncancel (also setting given to false) in the gen, then the alloc would only know its a restart maybe by checking another field, like sim-worker. Now that could be done, and possibly used to tell it to go to the same worker. This could be useful, if checkpointing was used. But perhaps, the scheduling field should really tell us that its to be restarted.

I'll also look again at the 'pause' example, as this has obviously been considered before.

[jlnav]

Is this closed by #760 ?