Fast freq

[mlee94]

Linked Issue: #19

Updating historical inputs only for now. Have to think about how we handle the schema change.

L673 expects all self.table_columns ie. they aren't optional. Maybe set the additional columns as optional, have default values or implement the date logic (so it only searches for additional columns after they were added to schema).

Can't think of anything elegant yet or you've probably already had a think about this.

[nick-gorman]

Hi @mlee94,

Thanks for the PR! Really appreciate the engagement in maintaining Nempy. Apologies, probably something I should have noted in an issue myself, but there is already a branch with changes for the FFR markets (https://github.com/UNSW-CEEM/nempy/tree/very_fast_fcas). I've tested these changes for backward compatibility and have been waiting for the market to begin operation so I can test against real data.

I think we've made very similar changes so far. I've dealt with the extra columns by checking if they exist in the downloaded data.

Happy for you to look at my changes, review them for potential issues, use them to inform your PR, and then I can review the PR and we can go from there? The only extra thing I'd add is it'd be good to do testing on a week or so of data before we merge and release the changes.

Let me know your thoughts.

Cheers,
Nick

[mlee94]

Hi @nick-gorman ,

I see you've already updated tests, and a few other things. So I've just merged changes from very_fast_fcas to fast_freq to catch any diffs (within our similar changes) but there weren't any.

[mlee94]

I'm happy to write a test on a week or so of data once AEMO updates the NEMDE XML inputs for October. Anticipating this will be mid next month.

[mlee94]

Tested on 100 random intervals between the start of the fast frequency response markets and end of Oct.

Summary of error in energy price across all regions:

• Mean price error: 0.17175635564022043
• Median price error: 0.0
• 5% percentile price error: -0.050982130191409604
• 95% percentile price error: 0.35580904152894177

Outputs:
benchmark_2023_ffr_random_intervals.csv

[nick-gorman]

Hi @mlee94,

Thanks for your work on this!

The changes look good to me.

I've also added a comparison of historical vs modelled FCAS prices to the benchmarking script, and the results for the fast frequency markets look about as good as the other FCAS markets or the energy markets. The summary of the results across all FCAS markets is:

• Mean price error: -0.02846662914326035
• Median price error: 0.0
• 5% percentile price error: -2.220446049250313e-16
• 95% percentile price error: 4.440892098500626e-16

With that covered off I'll merge the PR, and publish then a new version of Nempy once I've merged in some of the changes from other recent PRs.

Cheers,
Nick

[mlee94]

Nicely done!