Update for ATMS for End-to-End Test #757

emilyhcliu · 2023-11-20T19:39:23Z

List of items to be added in this PR:

BUFR Converter: Add YAML for converting ATMS BUFR to IODA
- The ObsValue is antenna temperature
- Input BUFR is normal feel
Testing YAML: re-evaluation results due to update CRTM from v2.3.0 to v2.4.1-jedi.1
- atms_npp_noqc.yaml --- 100% replication
- atms_npp.yaml --- 100 % replication
- atms_n20_noqc.yaml --- 100 % replication
- atms_n20.yaml --- ~ 1100% replication: two channel 7 observations difference
Config YAML:
- add two read routine QC: data thinning and removal of data from scan edge
- atms_npp.yaml
- atms_n20.yaml
Paring PR in JCSDA UFO repository
- This PR also has a paring PR in JCSDA UFO repository.
- It is OK to merge this one into GDASApp before the paring UFO PR
- The related test data (obs and geoval files from GSI) for UFO Evaluation have been updated (due to CRTM-2.4.1 update) on ORION and HERA.

Notes:

End-to-end testing completed without problem.
However, the filter results show one of the filters, 88-165 GHz scattering index check, tossed more data compared to GSI.
The excess screening from the 88-165 GHz scattering index check is under investigation and will be reported separately.
Please check comments below for validation results

…ioda.

…p_channels" for npp

emilyhcliu · 2023-11-20T19:54:46Z

UFO evaluation for CRTM-2.4.1 update

ATMS NPP
Results without QC
HofX Difference (No QC) Comparison (JEDI vs. GSI) for all channels

Initial Obs Error Difference (JEDI vs. GSI) for all channels (No QC)

Results with QC
HofX Difference with QC (JEDI vs. GSI) for all channels

Obs Erro Difference with QC

QC Difference (JEDI vs. GSI) for all channels

Note: 100% replication of GSI QC

emilyhcliu · 2023-11-20T19:56:18Z

UFO evaluation for CRTM-2.4.1 update

ATMS N20
Results without QC
HofX Difference (No QC) Comparison (JEDI vs. GSI) for all channels

Initial Obs Error Difference (JEDI vs. GSI) for all channels (No QC)

This is Initial Obs Error Assignment for ATMS N20

Results with QC
HofX Difference with QC Comparison (JEDI vs. GSI) for all channels

Obs Error Difference with QC Comparison (JEDI vs. GSI) for all channels

QC Difference (JEDI vs. GSI) for all channels

Notes: Two observations difference from channel 7

emilyhcliu · 2023-11-20T20:25:17Z

End-to-End (using ATMS NPP channel 2 as an example)

All Data

Comparison QC between UFO and GSI
The UFO (end-to-end) filters out more data. The scattering index check (88-165 difference) tosses more data.
(See the dark green color from the qc flag comparison below)

UFO after QC

GSI after QC

Comparison QC Flags between UFO and GSI
UFO QCFlags

GSI QC Flags

RussTreadon-NOAA

Ran 2021080100 gdas cycle from prepatmiodaobs through atmanlfinal. All jobs ran to completion. atms_n20 and atms_npp data processed. Increments look reasonable.

 0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-4.611234e-09 Max:+4.406635e-09 RMS:+6.948029e-12
  0: northward_wind                               | Min:-5.030879e-09 Max:+5.978815e-09 RMS:+7.534733e-12
  0: air_temperature                              | Min:-3.008589e-07 Max:+2.911516e-07 RMS:+1.419730e-10
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:-3.916608e-04 Max:+2.488543e-03 RMS:+7.406468e-07
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+1.618770e-20 RMS:+1.293217e-23
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+1.474788e-19 RMS:+2.167418e-22
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00

Final diagnostic tarball contains atms diagnostic files.

Orion-login-2:/work2/noaa/da/rtreadon/gdas-validation/comrot/gdas_eval_satwind_JEDI/gdas.20210801/00/analysis/atmos$ tar -tvf gdas.t00z.atmstat
-rw-r--r-- rtreadon/stmp 6888137 2023-11-20 20:17 diag_satwind_ahi_h8_2021080100.nc4.gz
-rw-r--r-- rtreadon/stmp 5797597 2023-11-20 20:17 diag_ascatw_ascat_metop-b_2021080100.nc.gz
-rw-r--r-- rtreadon/stmp 36257317 2023-11-20 20:17 diag_gnssro_2021080100.nc4.gz
-rw-r--r-- rtreadon/stmp 716249805 2023-11-20 20:19 diag_atms_npp_2021080100.nc.gz
-rw-r--r-- rtreadon/stmp  52721306 2023-11-20 20:19 diag_satwind_abi_goes-16_2021080100.nc.gz
-rw-r--r-- rtreadon/stmp  95641266 2023-11-20 20:20 diag_satwind_abi_goes-17_2021080100.nc.gz
-rw-r--r-- rtreadon/stmp   4746726 2023-11-20 20:20 diag_ascatw_ascat_metop-a_2021080100.nc.gz
-rw-r--r-- rtreadon/stmp 728609297 2023-11-20 20:22 diag_atms_n20_2021080100.nc.gz

Aside: as we validate more observation types, the run time for gdasatmanlfinal increases due to gzipping observation diagnostic files. With 8 files we are up to about 5 minutes wall time. We will likely need to parallelize this at a later date.

^[[38;21m2023-11-20 20:17:19,097 - INFO     - atm_analysis: Compressing 8 diag files to /work2/noaa/da/rtreadon/gdas-validation/comrot/gdas_eval_satwind_JEDI/gdas.20210801/00//analysis/atmos/gdas.t00z.atmstat.gz^[[0m
^[[38;5;39m2023-11-20 20:17:19,097 - DEBUG    - atm_analysis: Gzipping 8 diag files^[[0m
^[[38;5;39m2023-11-20 20:22:07,498 - DEBUG    - atm_analysis: Creating tar file /work2/noaa/da/rtreadon/gdas-validation/comrot/gdas_eval_satwind_JEDI/gdas.20210801/00//analysis/atmos/gdas.t00z.atmstat with 8 gzipped diag files^[[0m
^[[38;21m2023-11-20 20:22:10,568 - INFO     - atm_analysis: Copying /work/noaa/stmp/rtreadon/RUNDIRS/gdas_eval_satwind_JEDI/gdasatmanl_00/gdas.t00z.atmvar.yaml to /work2/noaa/da/rtreadon/gdas-validation/comrot/gdas_eval_satwind_JEDI/gdas.20210801/00//analysis/atmos^[[0m
^[[38;5;39m2023-11-20 20:22:10,569 - DEBUG    - atm_analysis: Copying /work/noaa/stmp/rtreadon/RUNDIRS/gdas_eval_satwind_JEDI/gdasatmanl_00/gdas.t00z.atmvar.yaml to /work2/noaa/da/rtreadon/gdas-validation/comrot/gdas_eval_satwind_JEDI/gdas.20210801/00//analysis/atmos/gdas.t00z.atmvar.yaml^[[0m

Approve.

RussTreadon-NOAA · 2023-11-20T20:08:12Z

parm/atm/obs/lists/gdas_prototype_3d.yaml

@@ -2,12 +2,16 @@ observers:
 ##- !INC ${OBS_YAML_DIR}/amsua_n19.yaml
 ##- !INC ${OBS_YAML_DIR}/sondes_prepbufr.yaml
 ##- !INC ${OBS_YAML_DIR}/atms_n20.yaml
+##- !INC ${OBS_YAML_DIR}/atms_npp.yaml


How do we want gdas-validation developers to update gdas_prototype_3d.yaml as they merge their gdas-validation work into feature/gdas-validation?

Add new observation types to the file but keep them commented out (inactive) ... as is done this PR Update for ATMS for End-to-End Test #757, or

Add new observations types to the file in active mode (don't comment them out) ... as is done in PR Update GNSSRO assimilation to end-to-end GDASApp validation #753.

don't change gdas_prototype_3d.yaml

Here is my two cents:
We can just leave satwind_g16 active as default and make the rest of the data type inactive.
The newly added data type will be inactive.

Works for me. As long as we all follow the same approach, we won't have any conflicts with gdas_prototype_3d.yaml. We can turn on all or a subset of the validated data once the sprint concludes.

emilyhcliu · 2023-11-20T20:46:21Z

@xincjin-NOAA and I will work on AMSU-A next...

emilyhcliu · 2023-11-20T20:49:04Z

The ATMS test data for UFO evaluation has been updated on both Orion and Hera.

CoryMartin-NOAA · 2023-11-20T21:28:20Z

The gdas_prototype_3d.yaml file has conflicts that must be resolved before this can be merged.

emilyhcliu · 2023-11-20T21:47:44Z

The gdas_prototype_3d.yaml file has conflicts that must be resolved before this can be merged.

Resolved!

In [ previous PR](#757) for ATMS, the end-to-end testing was done without data thinning. And the excessive QC filtering in clear-sky areas, compared to GSI, was found. (See [plots](#757 (comment))) This PR includes two fixes for ATMS in the end-to-end testing and one new feature: 1. fix the excess QC filtering in clear-sky areas. ---> This has a paring [UFO PR #3122](JCSDA-internal/ufo#3122) 2. add data thinning 3. add the diagnostic flags (QC) --- **This can [reproduce QC flags](#768 (comment)) from GSI in UFO** **To test updates in this PR, please check out the UFO branch: [feature/satrad](https://github.com/JCSDA-internal/ufo/tree/feature/satrad) from JCSDA-internal in gdas-validation. This branch consolidates all proposed code changes to UFO for gdas validation: UFO PR #JCSDA-internal/ufo#3122 UFO PR #JCSDA-internal/ufo#3121 UFO PR #JCSDA-internal/ufo#3094 --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

emilyhcliu added 4 commits November 20, 2023 13:12

Add atms YAML files under testing and config; add YAML for atms bufr2…

27c607f

…ioda.

Merge branch 'feature/gdas-validation' into feature/gdas-validation-atms

6eed4f1

Replace "all_channels" with "atms_n20_channels" for n20, and "atms_np…

a9866d3

…p_channels" for npp

Add atms_npp

f2446e0

ADCollard approved these changes Nov 20, 2023

View reviewed changes

RussTreadon-NOAA approved these changes Nov 20, 2023

View reviewed changes

Merge branch 'feature/gdas-validation' into feature/gdas-validation-atms

c73ba38

CoryMartin-NOAA merged commit 047e914 into feature/gdas-validation Nov 20, 2023
5 checks passed

CoryMartin-NOAA deleted the feature/gdas-validation-atms branch November 20, 2023 21:53

emilyhcliu mentioned this pull request Nov 27, 2023

A few updates for ATMS end-to-end testing #768

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update for ATMS for End-to-End Test #757

Update for ATMS for End-to-End Test #757

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023 •

edited

Loading

RussTreadon-NOAA left a comment

RussTreadon-NOAA Nov 20, 2023

emilyhcliu Nov 20, 2023

RussTreadon-NOAA Nov 20, 2023

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023

CoryMartin-NOAA commented Nov 20, 2023

emilyhcliu commented Nov 20, 2023

Update for ATMS for End-to-End Test #757

Update for ATMS for End-to-End Test #757

Conversation

emilyhcliu commented Nov 20, 2023 • edited Loading

emilyhcliu commented Nov 20, 2023 • edited Loading

UFO evaluation for CRTM-2.4.1 update

emilyhcliu commented Nov 20, 2023 • edited Loading

UFO evaluation for CRTM-2.4.1 update

emilyhcliu commented Nov 20, 2023 • edited Loading

End-to-End (using ATMS NPP channel 2 as an example)

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA Nov 20, 2023

Choose a reason for hiding this comment

emilyhcliu Nov 20, 2023

Choose a reason for hiding this comment

RussTreadon-NOAA Nov 20, 2023

Choose a reason for hiding this comment

emilyhcliu commented Nov 20, 2023 • edited Loading

emilyhcliu commented Nov 20, 2023

CoryMartin-NOAA commented Nov 20, 2023

emilyhcliu commented Nov 20, 2023

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Nov 20, 2023 •

edited

Loading