Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub Issue NOAA-EMC/GSI#344. Changes to monitor new observations from N21, G18, Himawari9 #345

Merged
merged 1 commit into from
Apr 13, 2022

Conversation

HaixiaLiu-NOAA
Copy link
Contributor

There are a list of new observations to be available in operational system from atms_n21, cris-fsr_n21, viirs-m_j2, ompsnp_n21, ompstc8_n21, gome_metop-c, abi_g18 and ahi_himawari9. Changes are made to fix files, src and scripts to monitor these new data. No impact on the analysis is expected from these changes.

@HaixiaLiu-NOAA
Copy link
Contributor Author

All 19 regression tests have been run. 18 passed. The test global_fv3_4denvar_C192 failed with an error message below.

The memory for global_fv3_4denvar_C192_loproc_updat is 3227036 KBs. This has exceeded maximum allowable hardware memory limit of 2516582 KBs,
resulting in Failure maxmem of the regression test.

@RussTreadon-NOAA
Copy link
Contributor

Questions / Comments

  • Should ompsnp_n21, ompstc8_n21, and gome_metop-c be added to util/Ozone_Monitor/nwprod/gdas_oznmon/fix/gdas_oznmon_satype.txt?
  • viirs-m_j2 has been added to global_satinfo.txt, but not /util/Radiance_Monitor/nwprod/gdas_radmon/fix/gdas_radmon_satype.txt. viirs-m_npp and viirs-m_j1 are also missing from gdas_radmon_sattype,txt. Are these omissions intentional or an oversight?

global_gsi.x was built from NOAA-EMC/GSI master at 47ee70b (control) and HaixiaLiu-NOAA/GSI master at fcae3f6 (test). A standalone gsi script was run using files from the operational 2022032300 gdas. The control and test runs generate b4b identical increment files. This is an expected result.

The test global_gsi.x ran 20 seconds longer than the control. This difference may be within normal run to run variability. It's possible, though, that the longer run time is due to additional warnings printed to fort.207 for the new monitored data. The operational abias file does not contain the new monitored data. radinfo_read notes this and prints a warning message.

@HaixiaLiu-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA , thank you very much for running independent tests to examine the impact of these changes. You are correct that the new sensors should be added into the util/oznmon. I will make those changes. For the viirs-m related entries, I will contact Xu Li (I guess those data are related to the nsst work) to see if they should be added into the satype list. Thank you again, Russ.

1 similar comment
@HaixiaLiu-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA , thank you very much for running independent tests to examine the impact of these changes. You are correct that the new sensors should be added into the util/oznmon. I will make those changes. For the viirs-m related entries, I will contact Xu Li (I guess those data are related to the nsst work) to see if they should be added into the satype list. Thank you again, Russ.

@RussTreadon-NOAA
Copy link
Contributor

When we update global_convinfo.txt, global_ozninfo.txt, global_satinfo.txt, or prepobs_errtable.global, we usually update files in or add files to fix/gfsv16_historical. Retrospective parallels often use fix files from fix/gfsv16_historical. This PR adds new data in monitor mode. Thus, it may not be necessary for this PR to update or add files to fix/gfsv16_historical. It's important to review this point prior to starting GFS v16.x retrospective parallels.

We have multiple GFS V16.x PRs altering fix files. For example, this PR, #345, and PR #343 both alter fix files. PR #343 was merged into the master today (3/23/2022). PR #343 updated global_convinfo.txt and prepobs_errtable.global. These changes do not conflict with this PR. Still care should be taken to ensure the final fix in NOAA-EMC/GSI master contains all GFS v16.x fix changes. It's easy for a desired / required change to be missed or undone.

@RussTreadon-NOAA
Copy link
Contributor

Merger of PR #338 into the master conflicts with changes in this PR to util/Radiance_Monitor/nwprod/gdas_radmon/fix/gdas_radmon_satype.txt.

Changes to fix from PR #343 are also being flagged as being in conflict with this PR. This does not appear to be a conflict in the sense that the same file(s) have been updated. Rather, the master fix submodule has advanced beyond the fix submodule in the developer branch.

ahi_himawari8 0.00 1.00 90
ahi_himawari9 0.00 1.00 90
atms_n21 -52.393 1.10 96
cris-fsr_n21 -48.130 3.333 30
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA Mar 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question regarding addition of viirs to gdas_radmon_satype.txt already mentioned in this PR. Do similar entries need to be added to gdas_radmon_scaninfo.txt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we update global_convinfo.txt, global_ozninfo.txt, global_satinfo.txt, or prepobs_errtable.global, we usually update files in or add files to fix/gfsv16_historical. Retrospective parallels often use fix files from fix/gfsv16_historical. This PR adds new data in monitor mode. Thus, it may not be necessary for this PR to update or add files to fix/gfsv16_historical. It's important to review this point prior to starting GFS v16.x retrospective parallels.

We have multiple GFS V16.x PRs altering fix files. For example, this PR, #345, and PR #343 both alter fix files. PR #343 was merged into the master today (3/23/2022). PR #343 updated global_convinfo.txt and prepobs_errtable.global. These changes do not conflict with this PR. Still care should be taken to ensure the final fix in NOAA-EMC/GSI master contains all GFS v16.x fix changes. It's easy for a desired / required change to be missed or undone.

I will carefully examine the fix directory to make sure this PR have the changes from PR #343

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good. Some questions / comments posted in PR. The NOAA-EMC/GSI master was updated 3/23/2022 with several PRs. This PR needs to be updated to the new head of the GSI master.

I am not familiar with EFSOI. The changes to EFSOI files in this PR look fine but it would be good to get confirmation from @AndrewEichmann-NOAA .

@HaixiaLiu-NOAA
Copy link
Contributor Author

Changes look good. Some questions / comments posted in PR. The NOAA-EMC/GSI master was updated 3/23/2022 with several PRs. This PR needs to be updated to the new head of the GSI master.

I am not familiar with EFSOI. The changes to EFSOI files in this PR look fine but it would be good to get confirmation from @AndrewEichmann-NOAA .

Good suggestion. Thank you, Russ. Mike, please ask @AndrewEichmann-NOAA and @CatherineThomas-NOAA as reviewers for the changes related to the EFSOI part.

@RussTreadon-NOAA
Copy link
Contributor

PRs #309, #338, and #343 were merged into the NOAA-EMC/GSI master on 3/23/2022. HaixiaLiu-NOAA/GSI master is 6 commits behind the authoritative master. The forked master must be brought up to date with the authoritative master before this PR can move forward.

@HaixiaLiu-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA My forked master was brought up to date with the authoritative master this morning with @MichaelLueken-NOAA 's help.

@RussTreadon-NOAA
Copy link
Contributor

Update to head of authoritative master and forked master. Recompile global_gsi.x and rerun 2022032300 gdas case. Two executables generate b4b identical results. These results differ from previous run due to changes in PR #343.

Note that new runs are 10-20 seconds longer than previous runs. Consistently see forked master run about 20 seconds longer than authoritative master. Not sure why this is the case since fork is not processing more data.

Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't comment on EFSOI changes but they seem ok. Other changes look good. Don't have any new data to confirm that code and fix files behave as expected.

@HaixiaLiu-NOAA
Copy link
Contributor Author

Thank you @RussTreadon-NOAA for reviewing this PR. I have contacted @AndrewEichmann-NOAA to get him review this PR as well.

@AndrewEichmann-NOAA
Copy link
Contributor

@HaixiaLiu-NOAA I don't anticipate the EFSOI changes being a problem.

@HaixiaLiu-NOAA
Copy link
Contributor Author

@AndrewEichmann-NOAA Thank you for confirming.

@MichaelLueken
Copy link
Contributor

@HaixiaLiu-NOAA I'm planning on submitting this PR to the review committee tomorrow morning after merging the next update to the authoritative repository. However, there is currently a conflict with the fix submodule that will need to be corrected before this work can be submitted. To correct the fix submodule:

  1. git clone --recursive git@github.com:HaixiaLiu-NOAA/GSI.git update
  2. cd update
  3. git remote add upstream https://github.com/NOAA-EMC/GSI.git
  4. git remote update
  5. git rebase upstream/master
  6. cd fix
  7. git checkout rev2
  8. git checkout DA_GFSv16.x_NewObs
  9. git merge rev2
  10. git push origin DA_GFSv16.x_NewObs
  11. cd ..
  12. git add fix
  13. git rebase --continue
  14. git push origin master --force

With this, your fork's master will be up-to-date with the current authoritative repository and the conflict in fix should be corrected. If you have any questions or encounter issues, please let me know.

…r_n21, ompsnp_n21, ompstc8_n21, gome_metop-c, abi_g18, ahi_himawari9, viirs-m_j2
@HaixiaLiu-NOAA
Copy link
Contributor Author

@MichaelLueken-NOAA Thank you for the clear instruction. It is done.

@MichaelLueken MichaelLueken changed the title changes to monitor new observations from N21, G18, Himawari9 GitHub Issue NOAA-EMC/GSI#344. Changes to monitor new observations from N21, G18, Himawari9 Apr 13, 2022
@MichaelLueken
Copy link
Contributor

The due date for this update has past with no feedback from the review committee, so I will give final approval to these changes and merge them to the authoritative repository.

@MichaelLueken MichaelLueken merged commit 7201778 into NOAA-EMC:master Apr 13, 2022
AndrewEichmann-NOAA pushed a commit to AndrewEichmann-NOAA/GSI that referenced this pull request Jun 6, 2022
GitHub Issue NOAA-EMC#344. Changes to monitor new observations from N21, G18, Himawari9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

changes to include observations potentially available in 2022
4 participants