Kusama Era Point Distribution isn't Uniform

[wpank]

Opening this issue after noticing that era point distribution when validating on Kusama isn't uniform - there are some validators that seem to consistently earn double the amount of era points than others. This was brought to my attention after a community validator mentioned they consistently get more era points after auto restarting their nodes after 20 minutes. I've been doing the same on my validator, and also consistently notice more era points when restarting the validator node every 10-20 minutes.

Right now I'm collecting more information on era points earned under different conditions (increasing peer count, restarting periods, machine type) and will update this issue with some evidence after looking at more eras and more conditions.

[rphmeier]

cc @andresilva

This might be better as a Substrate issue, since we don't do anything specifically related to points in this codebase.

[andresilva]

@wpank It should be possible to debug this by calling the babe_epochAuthorship RPC method during a given session and saving the result, while running the validator with -lbabe=debug logs for the duration of the session. We can then use these logs to verify that the node actually authored on all the slots that it was supposed to. You might not get rewards for all of the authored secondary slot blocks (since there might be a primary at the same slot and your block will then be re-orged out). But the node should try to author it regardless and we can double check that with the logs.

[andresilva]

We can also gather data on this from the validators stats page in polkadot.js app. You could then run the validator for a larger number of eras (let's say 3 days) without restarting, and then another 3 days with restarts every 20 minutes. Screenshots from 2 different validators:

We should get different results for the grey line trend if restarting the node really makes a difference. I have looked at some of the validators in the set who have been in the set for the past 80 eras and their era points average seems to converge to around ~305-310. There is a lot of variability in era points across eras though, and this will keep increasing as we add more validators to the set (since the epoch length stays the same). The more validators we have in the set the longer it will take for the average to converge.

If this is really happening then it must mean that somehow validators are not authoring on all the slots that they are allocated, and if this behavior is widespread then it would start affecting block times (as more validators miss their slots the block time increases), and it doesn't seem like we are observing such block time increases.

I am not discarding the possibility that this is happening, just saying that we're looking at a stochastic process so if we want to reason about averages we need to gather longer periods of data.

[rphmeier]

variance is nonzero but seems to average out