mode_analysis output query

[mpm-meto]

mode_analysis produces a .out and .txt files and also has the bycase option. This too produces two files. This is the .out file.
This output struck me as curious...

Fcst Valid Time Area Matched Area Unmatched # Fcst Matched # Fcst Unmatched # Obs Matched # Obs Unmatched
Jul 11, 2018 00:00:00 434 0 0 0 1 0
Jul 17, 2018 00:00:00 458 0 1 0 1 0
Jul 18, 2018 00:00:00 343 0 0 0 1 0
Aug 14, 2018 00:00:00 616 0 1 0 0 0
Aug 15, 2018 00:00:00 1888 0 1 0 1 0
Aug 16, 2018 00:00:00 1819 0 1 0 1 0
Aug 17, 2018 00:00:00 773 0 1 0 1 0

The # Fcst matched != # Obs matched, e.g. on Jul 11, or 18 or Aug 14 so how can those be matched? According to the .txt file these are not matched, so the columns in the .out file are, to me, not correctly populated.

For Jul 11th, the 434 should go into the area unmatched and the the ob count should be in the # Obs Unmatched column...
Ditto for Jul 18th, whereas for Aug 14th the value should be in the # Fcst Unmatched column.

job_summary_bycase_wind_speed_00Z_120_GM_vs_ANALYSIS_precipitation_amount_super.txt

This output was produced with MET11.0.0. It may be this was a known bug and has been fixed. I am yet to try running MET12.0.0 mode_analysis with the MET11.0.0 generated MODE output to see if that changes things. It might not. There may also be backward compatibility issues. I will update when I have tried this.

[JohnHalleyGotway]

@mpm-meto, thanks for sending along a sample data file to demonstrate.

I ran the following command to recreate the output you describe:

mode_analysis -lookin job_summary_bycase_wind_speed_00Z_120_GM_vs_ANALYSIS_precipitation_amount_super.txt -bycase -single -cluster

I'd say there are 2 main questions here:

1. How was this input to MODE-Analysis generated?
2. Is the output from MODE-Analysis consistent with the inputs given to it?

For (1), the output file name containing data for multiple times and the file name "job_summary" indicates that this is NOT output from MODE itself. Instead, I suspect you ran MODE for multiple valid times, and then used mode_analysis to run a -summary job, and this is output from that job. I'd recommend running mode_analysis only on output directly from mode itself. As you've found, running mode_analysis on its own output may produce confusing or inconsistent results.

For (2), yes, I think the output from mode_analysis is consistent with the input.

• We get output for each of the 7 valid times found in the input.
• All of the AREA is reported in the AREA MATCHED column because the input file only has lines for cluster objects, which are matched by definition.
• The total count of objects = 11, which is consistent with the 11 lines of input data.

So I'd say there is no problem with the output from MODE-Analysis, the problem lies in the input, which has clearly already been filtered.

I'll note that the mode_summary.R script provides output that is largely equivalent to MODE-Analysis. Although we would like to eventually replace all of the R-scripts in the MET repo with Python equivalents.

Hope that helps clarify.

[mpm-meto]

Thanks for checking @JohnHalleyGotway. This is reassuring. If it's down to user error, I'll own it, but I don't claim to understand where I've gone wrong. This is quite old output that I am revisiting. It's output from the prototype version of the MvMODE available in MET11. The snag may be just that... I was running mode_analysis on output from MODE run on the super objects... I will retrace my steps and see where that gets me. Sadly I can't go back and run anything MET11 based... the script I ran is attached, and the .out file output I pasted above and .txt file I attached were created by this script. I will modify the attached to use MET12 version, and I get the same output!
run_mode_analysis.sh.txt

[mpm-meto]

Just coming back to the bycase .out and .txt files. I am still confused by the column headings and the values in them.
I've not finished my investigations but taking Aug 17 and collating the .out and .txt, the 773 listed under "area matched" is in fact the total area (fcst + ob), not the matched area. There are also two matched pairs on this day but the bycase files pick up one. There is also a CF002_CO002. Every other check looks consistent. I've even checked that the total area = symmetric difference + 2 * intersection_area and all that checks out. I am just missing objects.... the values in the file are correct if there were just the one object but there are two objects. Either way the area referred to is not the matched area but the total.

I've only checked a case where the number of objects is the same in the forecast and ob fields. I haven't yet checked a mismatched scenario.

Running with MET12 made no difference to the results.

[mpm-meto]

I've attached this plot to show why the output is counter-intuitive. This is plotting three of these files using the annotation provided, i.e. "area matched" and then the columns of #fcst matched and #obs matched. In my file the unmatched columns only contain 0s. I have annotated the bars with whether the number of matched objects is the same (=), not the same (<>) and where one of fcst/ob has no objects (0). The fact that this is the case causes an anomaly because how can there be a matched area if there is no object, in one or the other...? So, whilst the txt files are correct the info abstracted into the bycase out file is not.

The "area matched" column actually contains the total area, not the matched area, and that to me is right, but wrong column heading. I think it is only picking up ONE object pair though, and that, I think, is when you impose a minlat/maxlat, minlon/maxlon on the extraction, and the filter does not appear to be applied consistently to all the output file types. It can lead to some anomalies.

I tested that by rerunning with a smaller region. This leads to a mismatch on Aug 14th (2018). There are 4 matched object pairs in the _pc.txt file, but in the forecast only .txt file there is just the one, and there are no observed objects in the equivalent observed .txt file. This leads me to conclude that whilst the fcst centroid fo CF001 falls inside the bounds, the matching CO001centroid does not. The pc file does not contain the info about the centroid positions, so how can it know what is inside the bounds or not? Hence it shows ALL FOUR. The "area matched" for this day only gives the CF001 area because that's the only part of the total area it has.
I have an excel spreadsheet (which I resorted to!) to make sense of this. Either way, I think there is something in the logic for abstraction that can lead to inconsistencies.... and the heading for column to in the bycase.out file needs to change.
mode_analysis_tests.xlsx