-
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Metrics and Simple Baselines #1
Comments
@peterdudfield @dantravers Any other metrics or simple baselines I should include? |
some discussion in here - https://docs.google.com/document/d/1E9pccSVVIfn8m14fUqBCVLWKNiU1dUe_zgTcurqLWww/edit Probably good to include
There's are lots ofthers, but making a v1 first would be good, then we can slow add to them. Good to make it module becasue of this Also: I'm not sure if nowcasting_utils is the right place, i think all that code is quite out of data. I'd be tempted to make a new repo, |
btw: thanks for making the issue @jacobbieker |
Yeah, sounds good! Just made the repo, will move this over |
Good use of 'transfer' feature |
Sounds good! Another "model" I would run is to compare against is PV_Live intraday versus PV_Live updated. This gives an estimate of accuracy for national and GSP that we know we want to beat. To generalise the "large errors" - for National I think this is good to look at errors > 1GW (or maybe 2GW). For site level or GSP level, I would apply a statistical measure. Propose to count the % of errors which are greater than 1.65sigma, where sigma is the standard deviation of the timeseries of outturns. 1.65sigma equates to the 5 / 95% range of outturns for the site. |
Okay, sounds good, I've added those. How much intraday PV Live do we have @peterdudfield ? Don't remember when we started collecting it |
We avtaully have a few years worth, Jame re run some things. I would suggestion we use Plive intraday as another baseline mode, rather than intagle each model with PV Live intraday. That make sense? |
Perhaps a straigh forward threshol could be v0, then for v1 we could look at something a bit more eloborate |
Yeah, that's what I was thinking for the model. We can compute the errors and save those too if we want to, but would do that later. Comparing to the day after PV Live is already what the other error metrics do so don't think we need to include that separately. Is the intraday saved somewhere in a file?
|
Thinking about this further - I would simplify and make the large errors anything above a % of the capacity for that region / site. E.g. 10% of installed capacity. |
I would suggest the following metrics are the "headline" metrics that we use to compare models at the first pass, and then look at others for more detail:
For national forecasts:
For site-level forecasts:
The errors by time of day, season, etc, are useful, and should be used for more detailed comparisons of models. It would be good to standardise on a way to present these metrics. I.e. a particular grid format. |
We should have standard metrics and very simple baselines for use in training our models.
Baseline Models:
Metrics:
The text was updated successfully, but these errors were encountered: