Create "Headline Metrics"

[jacobbieker]

I would suggest the following metrics are the "headline" metrics that we use to compare models at the first pass, and then look at others for more detail:

• nMAE
• split by forecast horizon

For national forecasts:

• % errors > 1GW
• MAE (although doesn't provide more information than nMAE, it is humanly understood)
• split by forecast horizon.

For site-level forecasts:

• % errors > 10% of installed capacity (large errors - an equivalent of the 1GW for national)

The errors by time of day, season, etc, are useful, and should be used for more detailed comparisons of models. It would be good to standardise on a way to present these metrics. I.e. a particular grid format.

Originally posted by @dantravers in #1 (comment)

[dantravers]

Two additions to the above:

1. We shoudl track MBE (mean bias error), as we have seen it non-zero at times.
2. The "step change", or "ramp rate" has been described by some clients as a key metric they look at. Client energy confirmed our understanding as below. Can we add this in to the metrics?
   If we are currently at 12 midday, and forecast for 13.00 is x,
   Then at 12.30, the forecast for 13.30 is y.
   Then the step change is y-x.
   This is compared to the figure of (actual @ 13.30 – actual @ 13.00).
   The above is the step change for 1 hour horizon. It can be measured for every horizon.

@dfulu, @peterdudfield - would be interesting to start to measure this on PVNet 1 / 2.

[peterdudfield]

Ive called MBE ME before, which is "mean error" compared to "MAE" - see here

Yea will have to think how to measure that live, maybe we get it in here first

[dantravers]

Update! Metric for ramp rate as described above, which we thought was non-standard, is in fact not the right definition. It should be the more standard:
If we are currently at 12 midday, and forecast for 13.00 is x, and forecast for 13.30 is y, then the forecast ramp is y-x.
This is compared to the figure of (actual @ 13.30 – actual @ 13.00).

[peterdudfield]

x = forecast, y = truth

x_{t+1} - x_{t} compared to y_{t+1} - y_{t}

[peterdudfield]

Outlook were focused on 1 to 2 hours