Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[End- to End Test code sprint] Add SEVIRI METEOSAT-8 and METEOSAT-11 to end-to-end testing #766

Merged
merged 44 commits into from
Mar 27, 2024

Conversation

azadeh-gh
Copy link
Collaborator

@azadeh-gh azadeh-gh commented Nov 22, 2023

(#765)

End-to-end testing for Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 and to METEOSAT-11

New files include:

  • parm/ioda/bufr2ioda//bufr2ioda_sevcsr.json : JSON containing data format, sensor/bufr2ioda_sevcsr.json, and satellite information
  • ush/ioda/bufr2ioda/bufr2ioda_sevcsr.py : bufr2ioda python code for converting SEVIRI METEOSAT-8 and METEOSAT-11 from BUFR to IODA
  • parm/atm/obs/config/seviri_m08.yaml.j2
  • parm/atm/obs/config/seviri_m11.yaml.j2
  • parm/atm/obs/lists/gdas_prototype_3d.yaml.j2

RussTreadon-NOAA and others added 20 commits November 9, 2023 13:48
Two updates for GOES AMV

1. BUFR Converter:
- remove wind direction and speed
- change sensorZenithAngle to satelliteZenithAngle
2. UFO Filters:
- change `sensorZenithAngle` to `satelliteZenithAngle`
- add `linear obs operator` section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no linearized component for applying the 10-meter factor.

The increments look reasonable:
```
0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-1.642335e-01 Max:+1.660654e-01 RMS:+2.746685e-04
  0: northward_wind                               | Min:-2.120068e-01 Max:+1.663539e-01 RMS:+2.631295e-04
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------
```

The UFO filter results vs. GSI  -- looks good


![ufo_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/017aeabc-d6ca-4643-b037-16f46dde76a0)


![gsi_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/d99599a2-3f36-4ef6-aa2e-4611500b8c43)
Rather than each new ob type being manually added to the list, we can
use `glob.glob` to get the list of available observation types.
Two updates for MetOp SCATWIND; one minor updates for GOES AMV 

1. BUFR Converter:
    - remove wind direction and speed
    - add `stationElevation` as zero 

2. UFO Filters:
    - add linear obs operator section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no
      linearized component for applying the 10-meter factor.
- add `Gaussian Thinning` filter to thin data with 75 km box (equal box
size, prefer data close to central of the box)

3. For GOES AMV YAML files, change output data extension from `nc4` to
`nc`

UFO vs. GSI O-F and QC comparison: O-F and data count passed look close
![ufo_ascat_metop-b_omf_windEastward_qc
(1)](https://github.com/NOAA-EMC/GDASApp/assets/36091766/f234543e-e0b5-4c6e-add9-3c7365c0c469)

![gsi_ascat_metop-b_omf_windEastward_qc](https://github.com/NOAA-EMC/GDASApp/assets/36091766/37f4fba8-bd2a-4a54-b45e-4055e3ef3142)

3Dvar increments are reasonable with MetOp-B SCATWIND alone.
```
0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-1.741092e+00 Max:+1.712287e+00 RMS:+2.282584e-03
  0: northward_wind                               | Min:-1.825957e+00 Max:+1.930385e+00 RMS:+2.496075e-03
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------

```

3DVar increments are also reasonable with MetOp-B + MetOp-A assimilated
…oda.x (#730)

This PR allows for all obtypes to be processed in two loops.

We still need to figure out a way to process these in parallel
(somewhat).
Using python multiprocessing to generate obs in parallel. The current
list of obs goes from 10+ minutes to completing in ~5.5 minutes.
@RussTreadon-NOAA noted that `obs linear operator` should be `linear obs
operator` in the GOES-17 satwind YAML. This fixes that.
This reduces runtime further to run everything in one parallel pool.
Now, things complete in under 4 minutes (as of now).
Adding satwinds from the Advanced Himawari Imager (AHI) from Himawari-8
to GDASApp end-to-end testing

new files include:
parm/atm/obs/config/satwind_ahi_h8.yaml: QC filter YAML for AHI
Himawari-8 satwinds
parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.json: JSON containing data
format, sensor, and satellite information for AHI Himawari-8 satwinds
ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.py: bufr2ioda code for
extracting AHI Himawari-8 satwinds from BUFR

modified files include:
ush/ioda/bufr2ioda/run_bufr2ioda.py: added `"satwind_amv_ahi"` to list
`BUFR_py`

See #741 for details on testing. JEDI/GSI comparisons look good with GSI
thinning turned off, but there are large ob-count disparities both
before and after QC when comparing GSI+thinning to JEDI+thinning. The
tested `Gaussian Thinning` filter is included in the YAML but commented
out with a note.

---------

Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>
Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
Instead of writing the filled out `.json` and `.yaml` files to
`$COM_OBS`, we write them to `$DATA` so that they get purged unless
`KEEPDATA="YES"`
Updating GNSSRO converter and adding the yaml file to GDASApp end-to-end
testing based on [#712](#712).

New file:
parm/atm/obs/config/gnssro.yaml: Added QC filters in the YAML

Updated files:
parm/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.json: Updated the satellite
information
ush/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.py: Updated pccf, qfro, and
satelliteAscending flag

Refer to [issue #750](#750)
for testing details. JEDI/GSI comparisons revealed a cutoff near
obs=0.03 Rad due to super refraction 2. The number of observations after
QC in JEDI is also inconsistent with GSI.
List of items to be added in this PR:
- BUFR Converter: Add YAML for converting ATMS BUFR to IODA
   - The ObsValue is antenna temperature
   - Input BUFR is normal feel 

- Testing YAML: re-evaluation results due to update CRTM from v2.3.0 to
v2.4.1-jedi.1
   - atms_npp_noqc.yaml --- 100% replication
   - atms_npp.yaml  --- 100 % replication
   - atms_n20_noqc.yaml --- 100 % replication
- atms_n20.yaml --- ~ 1100% replication: two channel 7 observations
difference
  
- Config YAML: 
* add two read routine QC: data thinning and removal of data from scan
edge
   - atms_npp.yaml
   - atms_n20.yaml   

- Paring PR in JCSDA UFO repository
- This PR also has a [paring PR in JCSDA UFO
repository](JCSDA-internal/ufo#3094).
   - It is OK to merge this one into GDASApp before the paring UFO PR
- The related test data (obs and geoval files from GSI) for UFO
Evaluation have been updated (due to CRTM-2.4.1 update) on ORION and
HERA.


Notes:
- End-to-end testing completed without problem. 
- However, the filter results show one of the filters, 88-165 GHz
scattering index check, tossed more data compared to GSI.
- The excess screening from the 88-165 GHz scattering index check is
under investigation and will be reported separately.
- Please check comments below for validation results
…o-end testing (#764)

Adding satwinds from the Spinning Enhanced Visible and InfraRed Imager
(SEVIRI) from METEOSAT-8 to GDASApp end-to-end testing

new files include:
parm/atm/obs/config/satwind_seviri_m8.yaml: QC filter YAML for SEVIRI
METEOSAT-8 satwinds
parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json: JSON containing
data format, sensor, and satellite information for SEVIRI METEOSAT-8
satwinds
ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.py: bufr2ioda code for
extracting SEVIRI METEOSAT-8 satwinds from BUFR

End-to-end testing results are provided in #758 - Note: Thinning filter
is provided in YAML but turned off due to irreconcilable differences
with GSI in testing.

---------

Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>
…765)

End-to-end testing for Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 and to METEOSAT-11
new files include:
parm/ioda/bufr2ioda//bufr2ioda_sevcsr.json : JSON containing data format, sensor/bufr2ioda_sevcsr.json, and satellite information
ush/ioda/bufr2ioda/bufr2ioda_sevcsr.py: bufr2ioda code for extracting SEVIRI METEOSAT-8 and METEOSAT-11 from BUFR
@RussTreadon-NOAA
Copy link
Contributor

Orion test
Install

  • parm/ioda/bufr2ioda/bufr2ioda_sevcsr.json
  • ush/ioda/bufr2ioda/bufr2ioda_sevcsr.py

in a working copy of feature/gdas-validation.

Copy seviri_m08.yaml and seviri_m11.yaml from parm/atm/obs/testing to /parm/atm/obs/config. Update both for use in g-w. It's probable that the updated seviri yamls require additional changes for thinning, qc, etc. This is just a starting point to allow fv3jedi_var.x execution.

Ensure seviri satbias, satbias_cov, and tlapse files are in appropriate COMROT directory. Add seviri_m08 and seviri_m11 to parm/atm/obs/lists/gdas_prototype_3d.yaml

Execute prepatmiodaobs, atmanlinit, and atmanlrun. The first two jobs run to completion. SEVIRI dump files are written to the appropriate COM_OBS directory

-rw-r--r-- 1 rtreadon da   6353993 Nov 22 11:07 gdas.t00z.seviri_m11.tm00.nc
-rw-r--r-- 1 rtreadon da   4862394 Nov 22 11:07 gdas.t00z.seviri_m08.tm00.nc

atmanlrun begins but fv3jedi_var.x fails with the exception

313: Exception:         Reason: An exception occurred inside ioda while opening a variable.
313:    name:   MetaData/sensorAzimuthAngle
313:    source_column:  0
313:    source_filename:        /work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/ioda/src/engines/ioda/src/ioda/Has_Variables.cpp
313:    source_function:        ioda::Variable ioda::detail::Has_Variables_Base::open(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &) const
313:    source_line:    108
313:
313: Exception: oops::Variational<FV3JEDI, UFO and IODA observations> terminating...

Notice that atms ioda format dump files contain MetaData float sensorAzimuthAngle(Location). This MetaData field is missing from seviri ioda format dump files. Should it be present?

@BrettHoover-NOAA
Copy link
Collaborator

I changed sensorZenithAngle to satelliteZenithAngle in my AMV code to match changes that @emilyhcliu had made - its possible that there could be a variable name mismatch between different parts of the code-base as well

Small code-change to include SEVIRI/METEOSAT-11 satwinds in end-to-end
validation. This uses the hook-ins already provided by the
SEVIRI/METEOSAT-8 satwinds code that was merged in
#764.

New file: parm/atm/obs/config/satwind_seviri_m11.yaml - this is nearly a
direct copy of parm/atm/obs/config/satwind_seviri_m8.yaml, since they
utilize the same filters. Only the header information has changed.

Modified file: parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json -
METEOSAT-11 was added to the `satellite_info` group

End-to-end testing looks very similar to what was seen for
SEVIRI/METEOSAT-8 satwinds, detailed in
#758. A brief run-down:

(LW)IR winds are in QC agreement down to a 0.02% difference
(cloud-top)WV winds are in QC agreement down to a 0.20% difference
VIS winds differ by 4.5%, but there are only 22 accepted winds in GSI
and 23 in JEDI, the difference is a single satwind

All ob, HofX, and ob-minus-HofX differences look good

Increment summary, from gdasatmanlrun.log:
```
  0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-3.479765e-01 Max:+3.430433e-01 RMS:+4.990993e-04
  0: northward_wind                               | Min:-4.226316e-01 Max:+3.376561e-01 RMS:+4.889429e-04
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+1.618770e-20 RMS:+1.293217e-23
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+1.474788e-19 RMS:+2.167418e-22
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------
```

Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>
@emilyhcliu
Copy link
Collaborator

@azadeh-gh There are 235 files changed. This is odd.

I think this needs a new merge. It says there are 235 files changed.

I just found the same thing. @azadeh-gh you changed 5 files, I remembered.

I changed the base from feature/gdas-validation to develop.
Now, the changes are down from 235 to 5.

Azadeh Gholoubi and others added 2 commits March 1, 2024 18:59
@RussTreadon-NOAA
Copy link
Contributor

Please update feature/Azadeh_gdas-validation with recent changes from GDASApp develop. Note that file parm/atm/obs/lists/gdas_prototype_3d.yaml conflicts with develop.

@RussTreadon-NOAA
Copy link
Contributor

@azadeh-gh , feature/Azadeh_gdas-validation is 47 commits behind develop. Updating with develop is straightforward with the following caveat.

File parm/atm/obs/lists/gdas_prototype_3d.yaml no longer exists in develop. The replacement file is named parm/atm/obs/lists/gdas_prototype_3d.yaml.j2. Your older copy of the file activates two observation yamls

- !INC ${OBS_YAML_DIR}/seviri_m08.yaml
- !INC ${OBS_YAML_DIR}/seviri_m11.yaml

These yamls are not listed in gdas_prototype_3d.yaml.j2. Adding them is not as simple as adding the above serviri yaml files. The above seviri yaml file need to be converted to jinja2 templates and then gdas_prototype_3d.yaml.j2 updated to include the seviri jinja2 templates.

Do you want assistance with this?

@azadeh-gh
Copy link
Collaborator Author

@RussTreadon-NOAA Do I need to update the seviri YAMLs to convert to jinja2?

@RussTreadon-NOAA
Copy link
Contributor

Yes, the seviri yamls need to be updated to jinja2 templates

  • replace $( foo ) with {{ foo }}
  • add single quotes around filenames on obsfile lines
  • add two blank spaces at the start of each line in the yaml file. shifting comments is options. The one exception to this is the first line. obs space shifts over two spaces as - obs space
  • change file suffix from .yaml to .yaml.j2

Beware: If you change the file suffix via git mv and in the same commit modify the new jinja2 template, the previous history of the yaml is lost. git treats the combination of these two actions as creation of a new file I think two commits will preserve the history. One commit modifies the file. Another commit moves the file from filename1 to filename2.

@azadeh-gh
Copy link
Collaborator Author

@RussTreadon-NOAA Thank you. I updated the YAMLS to jinja2.

@emilyhcliu
Copy link
Collaborator

emilyhcliu commented Mar 22, 2024

Reviewers. Please look at the T2O Issue #99 related to this PR for test results.

@azadeh-gh I have two comments:

  1. There are many plots in the issue #99
    I think it would be good to have a short summary of your validation results here.
  2. Please add error handling in the Python converter for missing input files or missing subsets
    • the need to add the error handling was identified by @RussTreadon-NOAA (see PR #937 for more infomation)
    • here is the example of error handling added in the converter

Screenshot 2024-03-21 at 11 34 43 PM

@RussTreadon-NOAA
Copy link
Contributor

@azadeh-gh , the conflict in parm/atm/obs/lists/gdas_prototype_3d.yaml.j2 is easily resolved. develop adds Emily's ozone work to the obs list. Your branch has seviri. Both can ozone and seviri can be in the obs list.

@azadeh-gh
Copy link
Collaborator Author

SEVIRI-QC-Flowchart pptx

Summary of validation results

No Thinning without QC (UFO vs. GSI) - Channel 5 Time Level 3

ufo_seviri_m08_omf_bc_channel5_time1
gsi_seviri_m08_omf_gsi_bc_channel5_time1

No Thinning with QC: UFO vs. GSI - Channel 5 Time Level 1

ufo_seviri_m08_omf_qc_channel5_time1
gsi_seviri_m08_omf_gsi_qc_channel5_time1

GSI_OmF_Seviri08_ch5_Thinned_Passed_QCs
JEDI_OmF_Seviri08_CH5_Thinned_PassedQCs

Here are the channel statistics plots (#obs, bias/std for OMF and final obs error) for JEDI and GDAS runs (Thinned)

O-F Bias/Standard Deviation

JEDI GDAS (original: ptime=0 & ithin_time=1)

Final Obs Error

JEDI GDAS (original: ptime=0 & ithin_time=1)

Number Obs Passed QC

JEDI GDAS (original: ptime=0 & ithin_time=1)

O-F Bias/Standard Deviation

JEDI GDAS (ptime=0 & ithin_time=2)

Final Obs Error

JEDI GDAS (ptime=0 & ithin_time=2)

Number Obs Passed QC

JEDI GDAS (ptime=0 & ithin_time=2)

@azadeh-gh
Copy link
Collaborator Author

@azadeh-gh , the conflict in parm/atm/obs/lists/gdas_prototype_3d.yaml.j2 is easily resolved. develop adds Emily's ozone work to the obs list. Your branch has seviri. Both can ozone and seviri can be in the obs list.

Thank you Russ. Updated the parm/atm/obs/lists/gdas_prototype_3d.yaml.j2 to have both SEVIRI and Ozone in the obs list.

@azadeh-gh
Copy link
Collaborator Author

Reviewers. Please look at the T2O Issue #99 related to this PR for test results.

@azadeh-gh I have two comments:

  1. There are many plots in the issue #99
    I think it would be good to have a short summary of your validation results here.

  2. Please add error handling in the Python converter for missing input files or missing subsets

    • the need to add the error handling was identified by @RussTreadon-NOAA (see PR #937 for more infomation)
    • here is the example of error handling added in the converter

Screenshot 2024-03-21 at 11 34 43 PM

Thank you for your suggestions.

  1. Added summary of our evaluation results from the Issue #99 here.
  2. Updated my python converter and added error handling for missing input files or missing subsets.

@RussTreadon-NOAA
Copy link
Contributor

@azadeh-gh , I went to test feature/Azadeh_gdas-validation for a near real-time case (2024022400). The sevcsr bufr dump file is zero length. Your code noted this and properly exited. That's good. A check of the gdas directory in the GDA shows all gdas.t*z.sevcsr.tm00.bufr_d files for 2024 are, thus far, zero length.

Do we know why all 2024 sevcsr dump files are zero length? Will the data return?

Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look reasonable. Unable to test in JEDI ATM CI case. Bufr dump file gdas.t00z.sevcsr.tm00.bufr_d is zero length.

-rw-r--r-- 1 wkolczyn global 0 Feb 24 05:53 /work/noaa/rstprod/dump/gdas.20240224/00/atmos/gdas.t00z.sevcsr.tm00.bufr_d

@ADCollard
Copy link
Collaborator

@RussTreadon-NOAA We cannot do a real time test of SEVIRI as the CSR stream has been discontinued. The same information is in the ASR stream, so the next step after this is to create an ASR converter.

Copy link
Collaborator

@ADCollard ADCollard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The results in JEDI-T2O Issue 99 look good. Approve.

@RussTreadon-NOAA
Copy link
Contributor

Thank you @ADCollard for explaining what I am seeing.

@CoryMartin-NOAA CoryMartin-NOAA merged commit 3ffc340 into develop Mar 27, 2024
5 checks passed
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/Azadeh_gdas-validation branch March 27, 2024 17:54
danholdaway added a commit that referenced this pull request Apr 8, 2024
* origin/develop:
  Use <filesystem> on a non c++17 supported machine (WCOSS ACORN) (#1026)
  Change generate_com to declare_from_tmpl (#1025)
  Commenting out more of the marine bufr 2 ioda stuff (#1018)
  make driver consistent with workflow driver (#1016)
  Update hashes now that GSI-B is working for EnVar (#1015)
  Add GitHub CLI to path for CI (#1014)
  Use _anl rather than _ges dimensions for increments in FV3 increment converter YAML (#1013)
  Fix inconsistent VIIRS preprocessing test (#1012)
  remove gdas_ prefix from executable filename in test_gdasapp_fv3jedi_fv3inc (#1010)
  Bugfix on Broken GHRSST Ioda Converter (#1004)
  Moved the marine converters to a "safe" place (#1007)
  restore ATM local ensemble ctest functionality (#1003)
  Add BUFR2IODA python API converter to prepoceanobs task (#914)
  Remove sst's from obs proc (#1001)
  JEDI increment write to cubed sphere history (#983)
  [End- to End Test code sprint] Add SEVIRI METEOSAT-8 and METEOSAT-11 to end-to-end testing (#766)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants