Define measure of persistence

[ppebay]

The main goal of this issue is to determine when it the "persistence" assumption (needed for statistically-based distributed LB) is satisfied so such LB can be efficiently performed

[ppebay]

For reference, coefficient of variance (CoV) calculated over sliding 4-long time windows, for 3 objects of the beta4pg-15 data exemplar:

[ppebay]

Same idea, but averaged out across all 64 objects, for 3-, 4-, and 10-long sliding windows:

Y-axis scale bounded above at 0.1 to prevent scale crushing by initial CoV values.

[ppebay]

@lifflander @PhilMiller @nlslatt

The Coefficient of Variation (CoV) appears to work well for our purpose, when computed over a sliding window. This is the ratio of stdev/mean (assuming mean >0 which shall always be the case for us as no object will have 0-time associated to it).

There is more “accumulated variation” as the size of the sliding window increases, which is to be expected, while at the same time the CoV becomes smoother. But in all case they seem to yield comparable results. So we should favor the smaller windows as they are faster to compute.

My proposal for a measure of persistence : coefficient of variation (which we can maintain locally on each object at very limited cost, say over a 4-long sliding window), averaged out across all objects each time we want to attempt a LB stage. If it’s greater than 10%, then pass (postpone) the LB. Maybe postpone it by at least the length of the sliding window, to avoid unnecessary global averaging.

[ppebay]

Quite a bit of research was done on this, 5 years ago, but it seems @lifflander that there isn't any longer a need to keep this open. We should probably write a short summary of the what was developed/tested (in case persistence-based LB is needed again) and close this one -- and transpose it into a vt issue.

[ppebay]

This is closed and superseded by vt #2303