is mean_skill working correctly with more than one model?

[daniel-caichac-DHI]

Hi all, I got a question because I am not sure I am getting sensible results when using mean_skill with more than one model.
To summarize:
I have two runs (of slighlty different lengths) , Run 000 with ~203,000 points and Run 023 with ~197,000 points
if I have a single comparer object (cc_aux) and get the mean_skill I get what I am supposed to get (I checked against the dataframe with all my PointObservations)

but If I add the comparer object of run 23 to the comparer of run 00 (with cc.add_comparer(cc_aux)) and then get the mean_skill my values change completely

Somehow the number of paired model-observations increases to ~8 million points
@ecomodeller : Is this a bug or am I not understanding how the mean_skill works?

[daniel-caichac-DHI]

Furthermore, when I have a single comparer and look at a single observation (either comparer of run 00 or run 23) I get ~3000 points

but when I do the same for the comparer with both runs, it all goes crazy and it creates ~114,000 points.

[daniel-caichac-DHI]

@jsmariegaard I think this is a bug...

[jsmariegaard]

Looks like it - could you make a small example that allows me to reproduce it?

[daniel-caichac-DHI]

@jsmariegaard
I think that I found the issue, and is related to the max_gap, as is apparently not implemented with the multi-model compare.
I am uploading:

• Some observed dfs0 data
• Some modelled dfs0 data at same location
• A very very small notebook to highlight the issue
  I had to zip it because otherwise I cannot upload it. A screenshot of the notebook is shown next.
  

If I remove the max_gap keyword then I get the same number of points in the single compare and multi-model compare skill (around ~40,000) , however as explained in that branch, I need to have max_gap implemented otherwise I get years of interpolated model data when modeling independent events (such as storms only)

Test_Data_multimodel.zip

[jsmariegaard]

I will have a look at your notebook today.

And if I am getting you right, this is only a problem in the https://github.com/DHI/fmskill/tree/max_gap_interp branch and associated #148 ?

[daniel-caichac-DHI]

If I run it with that PR yes, I get the issue.
I just updated that branch and then the kword in extract() changes from max_gap to max_model_gap but the issue remains

[jsmariegaard]

The issue seems to be related to add_comparer().

[daniel-caichac-DHI]

Yes

[daniel-caichac-DHI]

@ecomodeller Can we tag this as a bug??

[daniel-caichac-DHI]

I already did now.
the issue is the following:

• if I have 1 model I can add many observations with add_comparer(), no problem there
• but if I have 1 observation and many models, when I add the compares with add_comparer() some un-existent data is created.

See the error in the post above
#161 (comment)

[ecomodeller]

Bugs should be added to https://github.com/DHI/fmskill/issues

But I think it is being handled in #172