The different skill methods

[jsmariegaard]

There could be many ways to offer aggregated skill assessments in fmskill and to organize the methods and arguments. Here are my thoughts:

4 skill methods
I believe these four methods could cover most of our needs, and that the functionality and purpose of these are different enough to justify having four different methods:

• skill() - relevant in all cases, custom aggregation by df and by arguments; output is a table (df)
• mean_skill() - for ComparerCollection with different choices of weighting of observations; output is a table with one row per model
• temporal_skill() - rolling aggregations like Temporal skill assesment #11 e.g. for timeseries plotting; output is a time series
• spatial_skill() - mostly for altimetry data, x- y-binning of skill; output is xarray

All of them take the metrics argument.

Filter arguments
It should be easy to filter data upon making the skill assessment and the arguments should be the same as other Comparer methods such as scatter(). The filter arguments are:

• model (only if more than one model)
• observation (only ComparerCollection)
• start/end
• area

Later we will also add variable and maybe depth

Custom grouping
Each of the above methods will have good default grouping that fits most users. But they should also offer a by argument that will give more flexibility.

E.g. a monthly mean skill per model could be obtained by:

cc.skill(by=['model','freq:M'])

Together with the df argument the user has a lot of flexibility, e.g. to aggregate Hm0 results on wind direction bins:

df = cc.sel_df(observation='ABC')
df['wind_direction'] = wind_direction   # append first the wind direction to the all_df
cc.skill(df=df, by='wind_direction')

Other advanced assessments
IMO fmskill should do the most common skill aggregations for most users. It cannot and should not do everything for everyone. A user can always get the data and do more specialized analysis with pandas/xarray. If we have examples of such advanced usage we can always put them in a notebook.

[Hendrik1987]

Thanks for the clear summary! I agree with the main points.

It makes perfect sense to handle the custom grouping in the different skill methods to handle best fit defaults, as you point out.

However, I still prefer to outsource the 'filtering' or selection to a .sel() method returning a sub-set of a comparer. I think this makes sense, as the filtering will be the same for all skill and plotting methods - at least I can't think of anything else right now. #30

[jsmariegaard]

Thanks. You might be right about the sel() method - the main issue is copy/reference of data. The nice thing about sel_df() is that it returns a view of the dataframe, no copying is done. We can probably achieve the same with sel() but it's more complicated as it returns a whole Comparer object and we would like to avoid a deep copy. That is we I would like to postpone this a bit :-)

[jsmariegaard]

And furthermore, for most use (and for most users) I think the call should be as simple as possible. I think it's better to have:

cc.skill(model='SW_1')
cc.skill(start='2018-2-1')

Than:

cc.sel(model='SW_1').skill()
cc.sel(start='2018-2-1').skill()

[Hendrik1987]

I can see your points! I think my wish is also related to code readability and maintenance. E.g., add a new filter in one .sel() method vs. all methods, which take selection arguments. However, this does not concern the user and is not really an issue for now. So, happy to postpone this :-)

[ecomodeller]

All of this is implemented by now!