[End-to-End Test Code Sprint] Bufr to Ioda converter for SEVIRI METEOSAT-8 and to METEOSAT-11 #99

azadeh-gh · 2023-11-20T16:16:42Z

Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 and to METEOSAT-11 GDASApp end-to-end testing.

Create BUFR to IODA converter - python script
Create json file for python script
Update and add SEVIRI YAMLs to Config
Evaluate JEDI vs GSI OmF, QCs , and FinalObsError

azadeh-gh · 2024-01-12T16:34:43Z

GSI final obs error

JEDI final obs error

azadeh-gh · 2024-01-12T18:11:59Z

@ADCollard The plots show that both JEDI and GSI final obs error were almost 2.5 k everywhere. The min and max values on the plots are: vmin=2.5000, vmax=2.5001
GSI final obs error values:
4577 2.500105
4579 2.500102
4580 2.500095
4581 2.500106
4582 2.500103
9121 2.500106
9122 2.500101
9123 2.500099
9127 2.500100
9131 2.500108
Name: GsiFinalObsError, Length: 3967, dtype: float32

ADCollard · 2024-01-12T19:21:11Z

@azadeh-gh Thanks! I just spotted the "+2.5" on the color bar that is what was confusing me. Sorry for that. This looks good.

azadeh-gh · 2024-01-22T23:59:50Z

SEVIRI M11 channel 5 JEDI vs GSI final obs error

The values of obs error are 2.5 every where in both

emilyhcliu · 2024-03-04T16:29:35Z

@azadeh-gh

I worked on the seviri radiance a bit while I am in Tawian.
When I used the obs file generated from GSI diagnostic file and run end-to-end without thinning and with QC, good match was found between GSI and JEDI
When I used the obs file generated from BUFR and run end-to-end without thinning and with QC, good match was not found between GSU and JEDI. There were too much more data filtered out in in JEDI.

Three suggestions:
1. Something is wrong in QC in the end-to-end test. The problem may come from the MetaData, such as cloud information (cloud/clear amount) used by the QC procedure. It looks like you found the problem in the cloud/clear amount converted from BUFR. That's really good. Please test the updated IODA file (converted from BUFR) and run the end-to-end without thinning. The results should closely match GSI results.
2. Regarding observations (SECIRI radiances), the BUFR file contains 6 time levels. So, when you verify results, make sure you are validating the JEDI result with GSI for the same time level.
3. As for thinning, please see the attached flowchart. I moved the thinning to the fourth step (blue). It was in the second step before. If we place the thinning in the second step, more data will be screened out, we should execute thinning at the forth step. I tested thinning using the obs file generated from GSI diagnostic file. The results match well with GSI. I did not save the results. I will re-run the case and post the plots here.

azadeh-gh · 2024-03-04T16:48:34Z

@emilyhcliu Thank you Emily.

I regenerate the IODA un-thinned passed QC plots and add them here.
Then update the Thinning part in the YAML and update the thinned plots

emilyhcliu · 2024-03-04T17:33:44Z

Updated IODA vs GSI No thinning passed QCs

The number of obs passed QC does not match! Are these two plots from the same time level?

ADCollard · 2024-03-04T17:40:03Z

The distribution of QC'd obs is much better than before, but the O-Bs don't match particularly well either. So I am aslo guessing there is a time level issue.

emilyhcliu · 2024-03-04T17:55:28Z

These are results using IODA obs file generated from GSI diagnostic file

**No Thinning without QC (UFO vs. GSI) - Channel 5 Time Level 1 **

No Thinning with QC: UFO vs. GSI - Channel 5 Time Level 1

emilyhcliu · 2024-03-04T17:57:13Z

My color bar is too long. I should make it shorter (like the colorbar in your plots) so my globe will be bigger.
The results (after QC) should match this close! (15435 vs. 15350).

emilyhcliu · 2024-03-04T18:04:23Z

These are results using IODA obs file generated from GSI diagnostic file

Same as above, but for time level 3

**No Thinning without QC (UFO vs. GSI) - Channel 5 Time Level 3 **

No Thinning with QC: UFO vs. GSI - Channel 5 Time Level 3

emilyhcliu · 2024-03-04T18:10:25Z

This exercise of using IODA obs file from GSI diagnostics is to check if the QC filters work as expected without thinning.
Based on the results. We can be sure that the QC filters are correct! So, let's dig into the BUFR converter python code a bit.

emilyhcliu · 2024-03-04T18:12:56Z

Updated IODA vs GSI No thinning passed QCs

The number of obs passed QC does not match! Are these two plots from the same time level?

Your observation numbers are too big. I think you are plotting all-time levels. You need to separate the data into 6 time levels in your plotting code.

emilyhcliu · 2024-03-04T18:28:39Z

Here is the channel statistics after QC (unthinned) between GSI and UFO (using IODA obs file generated from GSI)

UFO and GSI match well for all channels

UFO Statistics (number of observation passed QC for each channel)

  0: QC seviri_m08 brightnessTemperature_4 : 16161 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_5 : 93326 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_6 : 23357 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_7 : 17378 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_8 : 17384 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_9 : 17195 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_10: 17009 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_11: 17339 passed out of 187230 observations.

UFO Statistics (number of observations passed QC for each channel) --- look at the 3rd column

 1578    4 seviri_m08        16152      0     -1.800  -1.5661124  -0.5673065   0.2080393   0.8670387   0.6556824
 1579    5 seviri_m08        92862   1446      2.500  -0.4410675   0.0869686   0.2364663   1.2157258   1.2126111
 1580    6 seviri_m08        23359      0      2.250  -0.0169656  -0.0179425   0.1018162   0.7874951   0.7872907
 1581    7 seviri_m08        17370      0     -1.250  -0.7939590  -0.0862284   0.1748905   0.6390446   0.6332003
 1582    8 seviri_m08        17378      0     -1.250  -1.0336799  -0.0956096   0.1803227   0.5686671   0.5605721
 1583    9 seviri_m08        17196      0     -1.250  -0.4881847  -0.0800431   0.2121771   0.6633558   0.6585089
 1584   10 seviri_m08        17000      0     -1.450  -0.5078334  -0.0538672   0.1964553   0.6989892   0.6969105
 1585   11 seviri_m08        17334      0     -1.250  -0.5422348  -0.0475899   0.1762843   0.5332766   0.5311488

ADCollard · 2024-03-04T23:09:01Z

@azadeh-gh It looks like there are still residual QC differences at the very bottom. Can you zoom in a report the QC flag there in the GSI?

azadeh-gh · 2024-03-04T23:24:45Z

@ADCollard The PreQC=55 on those locations.

emilyhcliu · 2024-03-04T23:27:25Z

@azadeh-gh amd @ADCollard
I found the problem in the Python code for the converter. I will update results and the fix here.

ADCollard · 2024-03-04T23:28:30Z

@azadeh-gh amd @ADCollard I found the problem in the Python code for the converter. I will update results and the fix here.

Nice @emilyhcliu . Is it still the cloud amount?

emilyhcliu · 2024-03-04T23:37:40Z

I checked the Python converter code. The fix for the cloud/clear amount is not correct. The cloud/clear amount should be one dimension (Location), not two dimensions (Location, Channel).

Here is the number of observations passed QC (unthinned) for m08 and m11 using the IODA obs file converted from BUFR (Python converter)

  0: QC seviri_m08 brightnessTemperature_4 : 16154 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_5 : 93326 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_6 : 23356 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_7 : 17373 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_8 : 17383 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_9 : 17195 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_10: 17011 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_11: 17339 passed out of 187225 observations.
---
  0: QC seviri_m11 brightnessTemperature_4 : 22476 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_5 : 102727 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_6 : 27377 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_7 : 23761 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_8 : 23798 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_9 : 23733 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_10: 23500 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_11: 23835 passed out of 197534 observations.

Here is the number of observations passed QC (3rd column; unthinned) for m08 and m11 from GSI:

 1578    4 seviri_m08        16152      0     -1.800  -1.5661124  -0.5673065   0.2080393   0.8670387   0.6556824
 1579    5 seviri_m08        92862   1446      2.500  -0.4410675   0.0869686   0.2364663   1.2157258   1.2126111
 1580    6 seviri_m08        23359      0      2.250  -0.0169656  -0.0179425   0.1018162   0.7874951   0.7872907
 1581    7 seviri_m08        17370      0     -1.250  -0.7939590  -0.0862284   0.1748905   0.6390446   0.6332003
 1582    8 seviri_m08        17378      0     -1.250  -1.0336799  -0.0956096   0.1803227   0.5686671   0.5605721
 1583    9 seviri_m08        17196      0     -1.250  -0.4881847  -0.0800431   0.2121771   0.6633558   0.6585089
 1584   10 seviri_m08        17000      0     -1.450  -0.5078334  -0.0538672   0.1964553   0.6989892   0.6969105
 1585   11 seviri_m08        17334      0     -1.250  -0.5422348  -0.0475899   0.1762843   0.5332766   0.5311488
---
 1602    4 seviri_m11        22486      0     -0.750  -2.3250160  -0.6533695   0.7378770   0.8289438   0.5101530
 1603    5 seviri_m11       102409   1911      2.500   0.2296573   0.0834692   0.2323322   1.2050544   1.2021602
 1604    6 seviri_m11        27390      0      2.250  -0.2998838  -0.0668661   0.1035764   0.7913032   0.7884730
 1605    7 seviri_m11        23766      0     -1.250  -0.6942221  -0.0758474   0.1457398   0.5892856   0.5843840
 1606    8 seviri_m11        23796      0     -1.250  -0.8387110  -0.1000189   0.1442385   0.5092437   0.4993249
 1607    9 seviri_m11        23741      0     -0.750  -0.4430402  -0.0774574   0.3777125   0.6373439   0.6326196
 1608   10 seviri_m11        23501      0     -0.800  -0.9435016  -0.0979794   0.4626198   0.6887142   0.6817091
 1609   11 seviri_m11        23832      0     -1.250  -0.9055599  -0.0906417   0.1152137   0.4306133   0.4209654

The numbers of observations passed QC (unthinned) match well between GSI and JEDI for all channels (active and passive)

I will post the plots for channel 5 here later (before and after QC)

azadeh-gh · 2024-03-04T23:40:18Z

@emilyhcliu but I checked the cloud amount values in bufr file and the cloud amount changes in each channel.
The values of cloud amount in the GSI file are not correct.

azadeh-gh · 2024-03-04T23:44:17Z

Here is the example of an observation that rejected with preQC=55 in GSI but passed in JEDI:
The value of ObsError is too big in GSI but 2.5 in JEDI

emilyhcliu · 2024-03-04T23:44:40Z

Here is the updated Python code on ORION:
/work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_sevcsr.py

I will leave the test with thinning to you. Please remember to move the thinning from the second step to the forth step as indicated in the updated flowchart

emilyhcliu · 2024-03-04T23:47:06Z

@azadeh-gh amd @ADCollard I found the problem in the Python code for the converter. I will update results and the fix here.

Nice @emilyhcliu . Is it still the cloud amount?

The dimension of the cloud/clear amount is not correct.

azadeh-gh · 2024-03-04T23:56:04Z

@emilyhcliu in the python script you changed this line: q.add("cloudFree", "/RPSEQ7/NCLDMNT")
to this: q.add("cloudFree", "/RPSEQ7{5}/NCLDMNT")
Do we need to read only channel 5 cloudAmount from the bufr file?

emilyhcliu · 2024-03-04T23:57:59Z

@azadeh-gh It looks like there are still residual QC differences at the very bottom. Can you zoom in a report the QC flag there in the GSI?

@azadeh-gh You have more data QCed out over sea-ice area. I think your O-Fs near the south pole must be very different from GSI. Please plot the O-F without QC (unthinned). If you see large differences between GSI and JEDI. The difference must be coming from HofX. So, compare the HofX between GSI and JEDI.

@emilyhcliu in the python script you changed this line: q.add("cloudFree", "/RPSEQ7/NCLDMNT") to this: q.add("cloudFree", "/RPSEQ7{5}/NCLDMNT") Do we need to read only channel 5 cloudamount from bufr file?

@azadeh-gh Yes, the cloudFree (clear fraction) should be read from channel 5.

emilyhcliu · 2024-03-05T00:00:30Z

@emilyhcliu but I checked the cloud amount values in bufr file and the cloud amount changes in each channel. The values of cloud amount in the GSI file are not correct.

@azadeh-gh In the IODA obs file from GSI (see below), the dimensions of cloud and clear amount are "Location" only/

       float cloudAmount(Location) ;
                cloudAmount:_FillValue = 9.96921e+36f ;
        float cloudFree(Location) ;
                cloudFree:_FillValue = 9.96921e+36f ;

  data:

   cloudAmount = 18, 40, 64, 56, 2, 0, 9, 3, 27, 43, 70, 0, 0, 0, 7, 31, 70,
      67, 75, 95, 0, 0, 0, 0, 12, 81, 53, 80, 77, 84, 80, 0, 2, 12, 15, 20,
      4, 15, 40, 39, 20, 30, 65, 0, 0, 0, 0, 0, 5, 0, 0, 6, 6, 0, 17, 43, 54,
      51, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 11, 5, 16, 18, 0, 0, 0, 0, 0, 0,
      0, 0, 0, 0, 4, 14, 8, 5, 8, 32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 4,
      6, 2, 18, 0, 0, 0, 0, 1, 0, 0, 0, 25, 54, 70, 49, 1, 2, 4, 9, 36, 53,

These are cloud amounts at each location without channel dependency.

emilyhcliu · 2024-03-05T00:05:23Z

@azadeh-gh Please try the updated python converter code in your end-to-end test for unthinned case and post your plots for channel 5 here.

I will post mine here for your reference.

emilyhcliu · 2024-03-05T00:08:26Z

Please note that there are 6 time levels. Please post seviri m08 channel 5 for time levels 1, 3, and 6.

azadeh-gh · 2024-03-05T00:48:46Z

@emilyhcliu Thank you, Emily. I will update the plots.
I wasn't aware that we're only reading and writing channel 5 cloud amount data from the BUFR file. Upon checking, noticed variations in cloud values across different channels.

emilyhcliu · 2024-03-05T00:51:51Z

Here are the results of the end-to-end for SEVIRI M08 Channel 5 (time levels 1, 3, and 6) using IODA obs file converted from BUFR

Unthinned without QC (UFO vs. GSI) --- time level 1

Unthnned with QC (UFO vs. GSI) --- time level 1

Unthinned without QC (UFO vs. GSI) --- time level 3

Unthnned with QC (UFO vs. GSI) --- time level 3

Unthinned without QC (UFO vs. GSI) --- time level 6

Unthnned with QC (UFO vs. GSI) --- time level 6

emilyhcliu · 2024-03-05T01:38:04Z

@azadeh-gh I updated the ratio of the plots (see plots above) so the globe is bigger.
The GSI and JEDI results match well.
I will stop here.
Please let me know if you run into any trouble with the thinned case.

ADCollard · 2024-03-05T15:08:38Z

@azadeh-gh @emilyhcliu I think you are both right. The confusion arises because in the GSI only channel 5 is used to generate the "cloud free" field even though the BUFR defines a cloud fraction for each channel. We (@azadeh-gh and me) were focussed on replicating the BUFR messages correctly and had not had not taken the second step yet.

        if(clrsky) then       
          call ufbrep(lnbufr,datasev1,1,ncld,iret,'NCLDMNT')
          rclrsky=bmiss
!          datasev1(1,5) is high-peaking water vapor channel
!          for SEVIRI CSR, clear-sky percentage are different between the high-peaking WV channel and other channels
           if(datasev1(1,5)>= zero .and. datasev1(1,5) <= 100.0_r_kind ) then
              rclrsky=datasev1(1,5)
!             first QC filter out data with less clear sky fraction
              if ( rclrsky < r70 ) cycle read_loop
           end if

Your original code did not specify a channel:

    q.add('cloudFree', '*/RPSEQ7/NCLDMNT')

which is why it was picking up the wrong channel in the original version. Emily explicitly adds the 5:

    q.add("cloudFree", "*/RPSEQ7{5}/NCLDMNT")

But (and this is my caveat), should the converter be producing what the GSI eventually wants (just channel 5) or should it properly replicate what is in the BUFR (channel dependent)?

I would argue for the latter (i.e., @azadeh-gh 's latest version of the converter and to explicitly specify channel 5 in that UFO YAML).

Thoughts?

ADCollard · 2024-03-05T15:38:42Z

One more thing. Looking in ObsSpace.yaml, cloudFree does appear to have dimensions of Location only.

- Variable: [ "cloudFree" ]
    Attributes: { units: "1" }

If we want to retain the original channel information we should use cloudAmount

Variable: [ "cloudAmount" ]
    Dimensions: [ [ "Location" ],  [ "Location", "Layer" ], [ "Location", "Channel"] ]
    Attributes: { units: "1" }

azadeh-gh · 2024-03-05T15:39:03Z

@emilyhcliu @ADCollard
I got my previous results even after using the new IODA observations.
This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same.
The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

azadeh-gh · 2024-03-05T16:55:17Z

@emilyhcliu @ADCollard
Here are the results from the updated Python code and YAML file. I was able to replicate Emily's plots for unthinned observations across all time levels, and I've included some examples here.
Subsequently, I added the thinning into the YAML file and generated corresponding plots.

Example of No thinning but passed QCs for time level 6, The observations that rejected in GSI but passed in JEDI had preQC= 5 (ifail_surface_qc=5, Reject due to not using over this surface in qc routine). a few obs rejected in JEDI but passed in GSI with EffectiveQC=19

Thinned and passed QCs

azadeh-gh · 2024-03-05T17:01:05Z

Channel 6

emilyhcliu · 2024-03-05T17:14:25Z

e

What we have if OK. I use cloudFree

@emilyhcliu @ADCollard I got my previous results even after using the new IODA observations. This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same. The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

@azadeh-gh, Two questions:
1, You got identical results for the runs using the original converter code (2D cloudAmount) and the updated code (1D cloudAmount) for channel 5?
2. For the unthinned case, did you modify the YAML (filters) for these two runs?

emilyhcliu · 2024-03-05T17:19:45Z

@azadeh-gh @emilyhcliu I think you are both right. The confusion arises because in the GSI only channel 5 is used to generate the "cloud free" field even though the BUFR defines a cloud fraction for each channel. We (@azadeh-gh and me) were focussed on replicating the BUFR messages correctly and had not had not taken the second step yet.
        if(clrsky) then       
          call ufbrep(lnbufr,datasev1,1,ncld,iret,'NCLDMNT')
          rclrsky=bmiss
!          datasev1(1,5) is high-peaking water vapor channel
!          for SEVIRI CSR, clear-sky percentage are different between the high-peaking WV channel and other channels
           if(datasev1(1,5)>= zero .and. datasev1(1,5) <= 100.0_r_kind ) then
              rclrsky=datasev1(1,5)
!             first QC filter out data with less clear sky fraction
              if ( rclrsky < r70 ) cycle read_loop
           end if
Your original code did not specify a channel:
    q.add('cloudFree', '*/RPSEQ7/NCLDMNT')
which is why it was picking up the wrong channel in the original version. Emily explicitly adds the 5:
    q.add("cloudFree", "*/RPSEQ7{5}/NCLDMNT")
But (and this is my caveat), should the converter be producing what the GSI eventually wants (just channel 5) or should it properly replicate what is in the BUFR (channel dependent)?

I would argue for the latter (i.e., @azadeh-gh 's latest version of the converter and to explicitly specify channel 5 in that UFO YAML).

Thoughts?

Both ways are OK with me.

azadeh-gh · 2024-03-05T17:54:53Z

e

What we have if OK. I use cloudFree

@emilyhcliu @ADCollard I got my previous results even after using the new IODA observations. This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same. The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

@azadeh-gh, Two questions: 1, You got identical results for the runs using the original converter code (2D cloudAmount) and the updated code (1D cloudAmount) for channel 5? 2. For the unthinned case, did you modify the YAML (filters) for these two runs?

Yes, but only for channel 5, because the values of cloudAmount were same.
Yes, after using your modified YAML and Python script got the identical results. Then added Thinning to the forth step of the updated YAML and plotted above plots.

emilyhcliu · 2024-03-05T18:11:37Z

e

What we have if OK. I use cloudFree

@emilyhcliu @ADCollard I got my previous results even after using the new IODA observations. This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same. The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

@azadeh-gh, Two questions: 1, You got identical results for the runs using the original converter code (2D cloudAmount) and the updated code (1D cloudAmount) for channel 5? 2. For the unthinned case, did you modify the YAML (filters) for these two runs?

Yes, but only for channel 5, because the values of cloudAmount were same.

Yes, after using your modified YAML and Python script got the identical results. Then added Thinning to the forth step of the updated YAML and plotted above plots.

I asked because the two versions of the converter create cloud amount in different dimensions. One of them has an additional channel dimension, so the YAML where the variable is used should be modified accordingly.
I am a bit puzzled. The YAML should be modified accordingly to use your original converter (2D cloud amount) correctly in the filters.

azadeh-gh · 2024-03-05T19:31:23Z

@emilyhcliu @ADCollard Updated YAMLs and python converter in PR#766

azadeh-gh · 2024-03-05T19:45:30Z

seviri_m11 Channel 5

seviri_m11 Channel 6

azadeh-gh · 2024-03-06T16:55:20Z

IODA vs GSI seviri_m08 Ch5 & Ch6

azadeh-gh · 2024-03-06T19:00:09Z

emilyhcliu · 2024-03-07T17:31:25Z

Here are the channel statistics plots (#obs, bias/std for OMF and final obs error) for JEDI and GDAS runs (Unthinned)

O-F Bias/Standard Deviation

JEDI	GDAS

Final Obs Error

JEDI	GDAS

Number Obs Passed QC

JEDI	GDAS

azadeh-gh · 2024-03-07T19:34:11Z

Thinned passed Qcs

JEDI
0: QC seviri_m08 brightnessTemperature_4: 546 passed out of 187225 observations.
0: QC seviri_m08 brightnessTemperature_5: 3986 passed out of 187225 observations.
0: QC seviri_m08 brightnessTemperature_6: 812 passed out of 187225 observations.
0: QC seviri_m08 brightnessTemperature_7: 629 passed out of 187225 observations.
0: QC seviri_m08 brightnessTemperature_8: 635 passed out of 187225 observations.
0: QC seviri_m08 brightnessTemperature_9: 616 passed out of 187225 observations.
0: QC seviri_m08 brightnessTemperature_10: 612 passed out of 187225 observations.
0: QC seviri_m08 brightnessTemperature_11: 630 passed out of 187225 observations.
0: QC seviri_m11 brightnessTemperature_4: 746 passed out of 197534 observations.
0: QC seviri_m11 brightnessTemperature_5: 4237 passed out of 197534 observations.
0: QC seviri_m11 brightnessTemperature_6: 922 passed out of 197534 observations.
0: QC seviri_m11 brightnessTemperature_7: 818 passed out of 197534 observations.
0: QC seviri_m11 brightnessTemperature_8: 822 passed out of 197534 observations.
0: QC seviri_m11 brightnessTemperature_9: 817 passed out of 197534 observations.
0: QC seviri_m11 brightnessTemperature_10: 808 passed out of 197534 observations.
0: QC seviri_m11 brightnessTemperature_11: 826 passed out of 197534 observations.

emilyhcliu · 2024-03-07T19:34:23Z

/Here are the channel statistics plots (#obs, bias/std for OMF and final obs error) for JEDI and GDAS runs (Thinned)

O-F Bias/Standard Deviation

JEDI	GDAS (original: ptime=0 & ithin_time=1)

Final Obs Error

JEDI	GDAS (original: ptime=0 & ithin_time=1)

Number Obs Passed QC

JEDI	GDAS (original: ptime=0 & ithin_time=1)

O-F Bias/Standard Deviation

JEDI	GDAS (ptime=0 & ithin_time=2)

Final Obs Error

JEDI	GDAS (ptime=0 & ithin_time=2)

Number Obs Passed QC

JEDI	GDAS (ptime=0 & ithin_time=2)

GSI (ptime=0 & ithin_time=1) --- original

 1578    4 seviri_m08          604      0     -1.800  -1.6664425  -0.6465958   0.2292820   0.9098325   0.6400851
 1579    5 seviri_m08         3967    110      2.500  -0.4593377   0.0501718   0.2999431   1.3692149   1.3682954
 1580    6 seviri_m08          892      0      2.250  -0.0446104  -0.0051772   0.1118269   0.8419722   0.8419563
 1581    7 seviri_m08          675      0     -1.250  -0.9303738  -0.1861693   0.2264663   0.7507246   0.7272746
 1582    8 seviri_m08          677      0     -1.250  -1.0797621  -0.1292236   0.2139787   0.6175371   0.6038654
 1583    9 seviri_m08          658      0     -1.250  -0.5937136  -0.1743215   0.2664909   0.7447730   0.7240848
 1584   10 seviri_m08          648      0     -1.450  -0.6084281  -0.1408040   0.2354207   0.7685143   0.7555054
 1585   11 seviri_m08          670      0     -1.250  -0.5726236  -0.0817513   0.2012528   0.5705738   0.5646868
--- 
 1602    4 seviri_m11          837      0     -0.750  -2.3486792  -0.5939860   0.7328441   0.8236663   0.5706197
 1603    5 seviri_m11         4253    158      2.500   0.3063720   0.1588176   0.2786130   1.3196343   1.3100426
 1604    6 seviri_m11         1059      0      2.250  -0.1432027   0.0857357   0.1072610   0.8230758   0.8185983
 1605    7 seviri_m11          912      0     -1.250  -0.7164994  -0.0783584   0.1766107   0.6654285   0.6607988
 1606    8 seviri_m11          916      0     -1.250  -0.8187622  -0.0682822   0.1578973   0.5309492   0.5265402
 1607    9 seviri_m11          913      0     -0.750  -0.4721010  -0.0954720   0.4703223   0.7222493   0.7159114
 1608   10 seviri_m11          903      0     -0.800  -0.9665710  -0.1124609   0.5617425   0.7783632   0.7701959
 1609   11 seviri_m11          921      0     -1.250  -0.8339662  -0.0163472   0.1259227   0.4512974   0.4510012

GSI (ptime=0 & ithin_time=2) --- closer to center of box and analysis time

 1578    4 seviri_m08          541      0     -1.800  -1.7938796  -0.7787300   0.2804260   1.0062840   0.6373280
 1579    5 seviri_m08         4006     73      2.500  -0.5140042  -0.0050128   0.2432654   1.2330830   1.2330728
 1580    6 seviri_m08          825      0      2.250  -0.0273681   0.0085220   0.0894444   0.7474635   0.7474149
 1581    7 seviri_m08          630      0     -1.250  -1.0174843  -0.2707135   0.2497690   0.7844096   0.7362150
 1582    8 seviri_m08          640      0     -1.250  -1.1344703  -0.1819824   0.2251281   0.6322797   0.6055246
 1583    9 seviri_m08          613      0     -1.250  -0.6675540  -0.2467316   0.2770614   0.7611647   0.7200661
 1584   10 seviri_m08          612      0     -1.450  -0.6967464  -0.2263676   0.2543705   0.7987981   0.7660523
 1585   11 seviri_m08          630      0     -1.250  -0.6514622  -0.1592031   0.2059898   0.5776025   0.5552288
 1602    4 seviri_m11          765      0     -0.750  -2.4577952  -0.6912033   0.8645461   0.8945727   0.5678894
 1603    5 seviri_m11         4305    102      2.500   0.2217094   0.0749274   0.2129663   1.1537406   1.1513050
 1604    6 seviri_m11          933      0      2.250  -0.2657382  -0.0308175   0.0848218   0.7280587   0.7274062
 1605    7 seviri_m11          826      0     -1.250  -0.8110140  -0.1725658   0.1776129   0.6605413   0.6376017
 1606    8 seviri_m11          829      0     -1.250  -0.8876646  -0.1335660   0.1586659   0.5307251   0.5136431
 1607    9 seviri_m11          827      0     -0.750  -0.5671903  -0.1910699   0.4541750   0.7173481   0.6914337
 1608   10 seviri_m11          821      0     -0.800  -1.0864129  -0.2276740   0.5498915   0.7728549   0.7385589
 1609   11 seviri_m11          834      0     -1.250  -0.9323592  -0.1171792   0.1180834   0.4373612   0.4213714

JEDI --- closer to center of box and analysis time

  0: QC seviri_m08 brightnessTemperature_4 : 549 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_5 : 3987 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_6 : 818 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_7 : 634 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_8 : 643 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_9 : 620 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_10: 617 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_11: 636 passed out of 187225 observations.
---  
  0: QC seviri_m11 brightnessTemperature_4 : 771 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_5 : 4299 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_6 : 936 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_7 : 842 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_8 : 847 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_9 : 842 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_10: 834 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_11: 852 passed out of 197534 observations.

azadeh-gh self-assigned this Nov 20, 2023

azadeh-gh changed the title ~~End-to-End Tests for SEVIRI [Bufr to Ioda converter]~~ End-to-End Test for SEVIRI [Bufr to Ioda converter] Nov 20, 2023

azadeh-gh changed the title ~~End-to-End Test for SEVIRI [Bufr to Ioda converter]~~ [End-to-End Test Code Sprint] Bufr to Ioda converter for SEVIRI METEOSAT-8 and to METEOSAT-11 Jan 10, 2024

azadeh-gh mentioned this issue Jan 24, 2024

[End-to-End Test Code Sprint] Bufr to Ioda converter for SEVIRI METEOSAT-8 and to METEOSAT-11 NOAA-EMC/GDASApp#765

Open

emilyhcliu mentioned this issue Mar 22, 2024

[End- to End Test code sprint] Add SEVIRI METEOSAT-8 and METEOSAT-11 to end-to-end testing NOAA-EMC/GDASApp#766

Merged

[End-to-End Test Code Sprint] Bufr to Ioda converter for SEVIRI METEOSAT-8 and to METEOSAT-11 #99

[End-to-End Test Code Sprint] Bufr to Ioda converter for SEVIRI METEOSAT-8 and to METEOSAT-11 #99

Comments

azadeh-gh commented Nov 20, 2023 • edited Loading

azadeh-gh commented Jan 12, 2024

GSI final obs error

JEDI final obs error

azadeh-gh commented Jan 12, 2024

ADCollard commented Jan 12, 2024 • edited Loading

azadeh-gh commented Jan 22, 2024 • edited Loading

SEVIRI M11 channel 5 JEDI vs GSI final obs error

emilyhcliu commented Mar 4, 2024 • edited Loading

azadeh-gh commented Mar 4, 2024

emilyhcliu commented Mar 4, 2024

Updated IODA vs GSI No thinning passed QCs

ADCollard commented Mar 4, 2024

emilyhcliu commented Mar 4, 2024

These are results using IODA obs file generated from GSI diagnostic file

emilyhcliu commented Mar 4, 2024 • edited Loading

emilyhcliu commented Mar 4, 2024 • edited Loading

These are results using IODA obs file generated from GSI diagnostic file

emilyhcliu commented Mar 4, 2024

emilyhcliu commented Mar 4, 2024

Updated IODA vs GSI No thinning passed QCs

emilyhcliu commented Mar 4, 2024 • edited Loading

Here is the channel statistics after QC (unthinned) between GSI and UFO (using IODA obs file generated from GSI)

ADCollard commented Mar 4, 2024

azadeh-gh commented Mar 4, 2024

emilyhcliu commented Mar 4, 2024

ADCollard commented Mar 4, 2024

emilyhcliu commented Mar 4, 2024 • edited Loading

azadeh-gh commented Mar 4, 2024 • edited Loading

azadeh-gh commented Mar 4, 2024

emilyhcliu commented Mar 4, 2024 • edited Loading

emilyhcliu commented Mar 4, 2024

azadeh-gh commented Mar 4, 2024 • edited Loading

emilyhcliu commented Mar 4, 2024

emilyhcliu commented Mar 5, 2024 • edited Loading

emilyhcliu commented Mar 5, 2024

emilyhcliu commented Mar 5, 2024

azadeh-gh commented Mar 5, 2024

emilyhcliu commented Mar 5, 2024 • edited Loading

Here are the results of the end-to-end for SEVIRI M08 Channel 5 (time levels 1, 3, and 6) using IODA obs file converted from BUFR

emilyhcliu commented Mar 5, 2024 • edited Loading

ADCollard commented Mar 5, 2024 • edited Loading

ADCollard commented Mar 5, 2024 • edited Loading

azadeh-gh commented Mar 5, 2024

azadeh-gh commented Mar 5, 2024 • edited Loading

Example of No thinning but passed QCs for time level 6, The observations that rejected in GSI but passed in JEDI had preQC= 5 (ifail_surface_qc=5, Reject due to not using over this surface in qc routine). a few obs rejected in JEDI but passed in GSI with EffectiveQC=19

Thinned and passed QCs

azadeh-gh commented Mar 5, 2024

Channel 6

emilyhcliu commented Mar 5, 2024 • edited Loading

emilyhcliu commented Mar 5, 2024

azadeh-gh commented Mar 5, 2024 • edited Loading

emilyhcliu commented Mar 5, 2024

azadeh-gh commented Mar 5, 2024

azadeh-gh commented Mar 5, 2024 • edited Loading

seviri_m11 Channel 5

seviri_m11 Channel 6

azadeh-gh commented Mar 6, 2024 • edited Loading

IODA vs GSI seviri_m08 Ch5 & Ch6

azadeh-gh commented Mar 6, 2024

emilyhcliu commented Mar 7, 2024 • edited Loading

azadeh-gh commented Mar 7, 2024 • edited Loading

Thinned passed Qcs

emilyhcliu commented Mar 7, 2024 • edited Loading

azadeh-gh commented Nov 20, 2023 •

edited

Loading

ADCollard commented Jan 12, 2024 •

edited

Loading

azadeh-gh commented Jan 22, 2024 •

edited

Loading

emilyhcliu commented Mar 4, 2024 •

edited

Loading

emilyhcliu commented Mar 4, 2024 •

edited

Loading

emilyhcliu commented Mar 4, 2024 •

edited

Loading

emilyhcliu commented Mar 4, 2024 •

edited

Loading

emilyhcliu commented Mar 4, 2024 •

edited

Loading

azadeh-gh commented Mar 4, 2024 •

edited

Loading

emilyhcliu commented Mar 4, 2024 •

edited

Loading

azadeh-gh commented Mar 4, 2024 •

edited

Loading

emilyhcliu commented Mar 5, 2024 •

edited

Loading

emilyhcliu commented Mar 5, 2024 •

edited

Loading

emilyhcliu commented Mar 5, 2024 •

edited

Loading

ADCollard commented Mar 5, 2024 •

edited

Loading

ADCollard commented Mar 5, 2024 •

edited

Loading

azadeh-gh commented Mar 5, 2024 •

edited

Loading

emilyhcliu commented Mar 5, 2024 •

edited

Loading

azadeh-gh commented Mar 5, 2024 •

edited

Loading

azadeh-gh commented Mar 5, 2024 •

edited

Loading

azadeh-gh commented Mar 6, 2024 •

edited

Loading

emilyhcliu commented Mar 7, 2024 •

edited

Loading

azadeh-gh commented Mar 7, 2024 •

edited

Loading

emilyhcliu commented Mar 7, 2024 •

edited

Loading