[Feature request] A second cross depending on the evaluations sum from all trials

[nikohansen]

When a solver terminates without relying on the experimentation budget, the number of independent trials is relevant for the number successes we expect to see. A single success is enough to solve the function+dimension via (simulated) restarts. For this reason, it would be useful know the sum of all evaluations from all trials for a given function+dimension.

Open questions: which marker should we use? What should we do if the sum is larger than the x-limit? Does it make sense to divide by the number of different instances too (a third symbol?)? Or does it make sense to divide by the number of successes too, that is, have an ERT marker?

[nikohansen]

Let runtime refer to successful and unsuccessful trials here, then we have the median of

• runtime for each trial/run (see also median budget cross goes astray with many repetitions #2287)
• sum of runtimes over the same instance
• sum of runtimes over all instances of a function+dimension
• sum of runtimes per success, aka ERT (different symbol)

The first three are monotonously increasing.

As we preferably make no distinction between instances and restarts, the sum over only the same instance does not give a preferable analytical viewpoint? As we want to replace within-trial restarts with additional trials, the second does give the single runtime (first bullet point) of the previous setup.

For one success, the last two are the same. This suggests to display the sum of runtimes divided by max(1, number of successes)?

EDIT: these symbols, except for ERT, are only useful if we have unsuccessful runs. Since recently, we display the max of successful and unsuccessful runs medians. This supports the idea to display the sum of runtimes divided by max(1, number of successes).