Lots of alerts about task that has ended unexpectedly

[eskimor]

On Kusama we get logs of alerts:

 Continuous task large-statement-fetcher on node kusama-validator-aws-ue1-1-int.aws.parity.io:9615 ended unexpectedly.

producing a lot of noise. I believe @tomaka mentioned that there is some heuristic to detect whether a task is supposed to stay running or is short lived. In this particular case the task is short lived (spawned to fetch some data, once it succeeds it will stop), so the heuristic seems to be wrong.

[gilescope]

Has this recently changed or been a constant niggle? I'm just wondering if we suddenly have a lot more large statements than usual (i.e. after we have rolled out some code change)?

[bkchr]

This has already been discussed multiple times, the problem is that the alerting rules live in the substrate repo: https://github.com/paritytech/substrate/blob/master/.maintain/monitoring/alerting-rules/alerting-rules.yaml

However, it doesn't make any sense because they are mainly polkadot specific. We should finally move them to polkadot and modify them properly.

[bkchr]

But this is really not Substrate code related.

[tomaka]

We could simply remove this alert altogether. As far as I can remember it has only once detected an actual issue with a task. All the other alert triggers have been false positives.

[eskimor]

Has this recently changed or been a constant niggle? I'm just wondering if we suddenly have a lot more large statements than usual (i.e. after we have rolled out some code change)?

That is actually a real good question. We should mostly have large statements on parachain runtime upgrades (or on large messages).