Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARW Trajectory Diagnostics #84

Merged
merged 6 commits into from
Jan 12, 2017
Merged

ARW Trajectory Diagnostics #84

merged 6 commits into from
Jan 12, 2017

Conversation

stacywalters
Copy link
Contributor

@stacywalters stacywalters commented Dec 27, 2016

TYPE: enhancement

KEYWORDS: trajectory

SOURCE: internal

DESCRIPTION OF CHANGES:

The original "trajectory" code in routine module_em.F only computed the trajectory
spatial coordinates per trajectory, per time step. The new code, in share/module_trajectory.F,
takes trajectory coordinate values and linearly interpolates in x,y, and z to get the
value of up 100 variables per trajectory per time step. The code will detect and handle
trajectory coordinate values that are between the grid parcels. The user may have
up to 1000 trajectories per domain and as mentioned up to 100 variables per trajectory.

Output goes to netcdf files wrfout_traj_.

LIST OF MODIFIED FILES:

M Registry/Registry.EM_COMMON
M Registry/Registry.NMM
M chem/chem_driver.F
M chem/depend.chem
M dyn_em/module_em.F
M dyn_em/solve_em.F
M dyn_em/start_em.F
M external/RSL_LITE/module_dm.F
M main/depend.common
M share/Makefile
M share/module_check_a_mundo.F
A share/module_trajectory.F
M share/solve_interface.F

TESTS CONDUCTED:

WTF reg test, version 3.04, has been successfully completed. The code has been tested in ACOM by
Mary Barth, Megan Bella, and Gabrielle Pfister.

KEYWORDS: trajectory

SOURCE: internal

DESCRIPTION OF CHANGES:

The original "trajectory" code in routine module_em.F only computed the trajectory
spatial coordinates per trajectory, per time step.  The new code, in share/module_trajectory.F,
takes trajectory coordinate values and linearly interpolates in x,y, and z to get the
value of up 100 variables per trajectory per time step.  The code will detect and handle
trajectory coordinate values that are between the grid parcels.  The user may have
up to 1000 trajectories per domain and as mentioned up to 100 variables per trajectory.

LIST OF MODIFIED FILES:

M	Registry/Registry.EM_COMMON
M	Registry/Registry.NMM
M	chem/chem_driver.F
M	chem/depend.chem
M	dyn_em/module_em.F
M	dyn_em/start_em.F
M	external/RSL_LITE/module_dm.F
M	main/depend.common
M	share/Makefile
M	share/module_check_a_mundo.F
A	share/module_trajectory.F
M	share/solve_interface.F

TESTS CONDUCTED:

WTF reg test, version 3.04, has been successfully completed. The code has been tested in ACOM by
Mary Barth,  Megan Bella, and Gabrielle Pfister.
@davegill davegill changed the title TYPE: enhancement ARW Trajectory Diagnostics Dec 27, 2016
@@ -438,6 +438,7 @@ module_sf_clm.o: module_cam_shr_kind_mod.o \
module_cam_shr_const_mod.o \
module_cam_support.o \
module_sf_urban.o \
module_ra_gfdleta.o \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is there a new dependency on module_ra_gfdleta.o?

@@ -606,6 +606,13 @@ SUBROUTINE check_nml_consistency
model_config_rec%num_traj = 0
END IF

#elif( NMM_CORE == 1 )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jamiebresch
Do we need a similar ifdef for 3dvar in check_a_mundo?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a "#endif" added before line 597 to conclude the EM_CORE test?
Then, start another ifdef for the trajectory thing?
if NMM_CORE == 1 or DA_CORE==1, no trajectory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jamiebresch
Has this ifdef issue in check_a_mundo been satisfactorially addressed? It seems like you wanted something like this:

#endif
#if ((EM_CORE == 1) && (DA_CORE != 1))
!----------------------------------------------------------------------------
! If trajectory option is turned off, make sure the number of trajectories is
! zero.
!----------------------------------------------------------------------------
   IF ( ( model_config_rec%traj_opt .EQ. 0 ) .AND. &
        ( model_config_rec%num_traj .NE. 0 ) ) THEN
         WRITE (wrf_err_message, FMT='(A,A)') '--- WARNING: traj_opt is zero, but ', &
                'num_traj is not zero; setting num_traj to zero.'
         CALL wrf_message ( wrf_err_message )
         model_config_rec%num_traj = 0 
   END IF

#else
!----------------------------------------------------------------------------
! If NMM core or DA, and trajectories are on, then halt.
!----------------------------------------------------------------------------
   IF ( model_config_rec%traj_opt /= 0 ) THEN
     call wrf_error_fatal( 'Trajectories not supported in NMM or DA' )
   END IF
#endif

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I tested it some days ago, 3DVAR and 4DVAR both ran fine. DA doesn't care about the traj_opt setting.
I just tested 4DVAR with stacy's latest TRJ branch but with an WRFPLUS that doesn't yet have the trajectory mods in start_em.F. It still runs fine with traj_opt=1.
The current TRJ branch looks OK.

@davegill
Copy link
Contributor

Stacy,

  1. There are no new variables in the Registry.EM_COMMON. If the output is in a netcdf file, where are these variables defined?
  2. Give an example of how you turn the option on/off, control which variables, starting and stopping time, initial locations, periodically follow same starting (i,j,k), etc.
  3. Please add a sample of what the netcdf -h returns for the new file.
  4. What happens to a trajectory during DFI (when the model runs backwards)?
  5. How about restarts? Is there any bit-for-bit results that you guys have manufactured, where a long-running long used trajectories?
  6. Are you able to get bit-wise identical answers using different core counts with MPI? Are you able to use OpenMP with multiple cores and get the same bit-identical answers as with a single processor?
  7. Since this is an additional output file and is not "gridded", how do you use this new file to work with trajectories?
  8. This generic info on trajectories could be put in a top-level README.trajectory, but let's wait for the rest of the WRF Dev to weigh-in on that idea (which can be added later, also).

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 27, 2016 via email

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 27, 2016 via email

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 27, 2016 via email

@jamiebresch
Copy link
Contributor

@stacywalters @davegill

  1. Is the variable no_trajectory defined?
  2. There might be various places that if DA_CORE!=1 needs to be added. I'll need some time to figure it out.

@davegill
Copy link
Contributor

davegill commented Dec 27, 2016 via email

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 28, 2016 via email

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 28, 2016 via email

@davegill
Copy link
Contributor

davegill commented Dec 28, 2016 via email

@jamiebresch
Copy link
Contributor

@stacywalters Oh, right. I was looking at WRFDA's version of frame/module_state_description.F which is different from ARW.

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 28, 2016 via email

@jamiebresch
Copy link
Contributor

@stacywalters From my limited tests, 3DVAR/4DVAR/WRFPLUS all ran fine with your TRJ branch code as it is.

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 28, 2016 via email

@stacywalters
Copy link
Contributor Author

stacywalters commented Dec 28, 2016 via email

@davegill
Copy link
Contributor

davegill commented Dec 28, 2016 via email

@dudhia
Copy link
Collaborator

dudhia commented Dec 30, 2016

In addition to a document, I think we could have a short README for this as well as something in examples.namelist. This can all be done later.

Copy link
Contributor

@mkavulich mkavulich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by wrf-dev committee contingent on one final bit-for-bit test with different core counts.


i_beg = max( 1,its-1 )
i_end = min( ite+2,ide-1 )
do j=max(1,jts-1),min(jte+2,jde-1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davegill Is this the right set of "mu"s to be using here?

i_end = min( ite+2,ide-1 )
do j=max(1,jts-1),min(jte+2,jde-1)
do k=kms,kme
u(its:i_end,k,j)=ru_m(its:i_end,k,j)/muu(its:i_end,j)*msfu(its:i_end,j)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stacy,
Change these three lines

  1. Make the "mu" variables upper case.
    muu -> MUU
    muv -> MUV
    mut -> MUT

  2. Split the loop into two pieces
    DO k = kms,kme-1
    the lines with u and v

DO k = kms,kme
the line with w is here

@davegill
Copy link
Contributor

davegill commented Jan 9, 2017

Stacy,
Also, Wei pointed out that the "mu" variables from the solver should be fixed.

muu -> muus
muv -> muvs
mut -> muts

Your line numbers may vary, but in dyn_em/solve_em.F, in the call to trajectory:

Original code

3469      !$OMP PARALLEL DO   &
3470      !$OMP PRIVATE ( ij )
3471        DO ij = 1 , grid%num_tiles
3472 
3473            call trajectory (grid,config_flags,                                     &
3474                             grid%dt,grid%itimestep,grid%ru_m, grid%rv_m, grid%ww_m,&
3475                             grid%mut,grid%muu,grid%muv,                            &

New code

3469      !$OMP PARALLEL DO   &
3470      !$OMP PRIVATE ( ij )
3471        DO ij = 1 , grid%num_tiles
3472 
3473            call trajectory (grid,config_flags,                                     &
3474                             grid%dt,grid%itimestep,grid%ru_m, grid%rv_m, grid%ww_m,&
3475                             grid%muts,grid%muus,grid%muvs,                            &

KEYWORDS: trajectory

SOURCE: internal

DESCRIPTION OF CHANGES:

The original "trajectory" code in routine module_em.F only computed the trajectory
spatial coordinates per trajectory, per time step.  The new code, in share/module_trajectory.F,
takes trajectory coordinate values and linearly interpolates in x,y, and z to get the
value of up 100 variables per trajectory per time step.  The code will detect and handle
trajectory coordinate values that are between the grid parcels.  The user may have
up to 1000 trajectories per domain and as mentioned up to 100 variables per trajectory.

Inputs are in the namelist files wrfinput_traj_d<domain>.
Outputs are in the netcdf files wrfout_traj_d<domain>.

LIST OF MODIFIED FILES:

M	Registry/Registry.EM_COMMON
M	Registry/Registry.NMM
M	chem/chem_driver.F
M	chem/depend.chem
M	dyn_em/module_em.F
M	dyn_em/start_em.F
M	external/RSL_LITE/module_dm.F
M	main/depend.common
M	share/Makefile
M	share/module_check_a_mundo.F
A	share/module_trajectory.F
M	share/solve_interface.F

TESTS CONDUCTED:

WTF reg test, version 3.05, has been successfully completed. The code has been tested in ACOM by
Mary Barth,  Megan Bella, and Gabrielle Pfister.
KEYWORDS: trajectory

SOURCE: internal

DESCRIPTION OF CHANGES:

The original "trajectory" code in routine module_em.F only computed the trajectory
spatial coordinates per trajectory, per time step.  The new code, in share/module_trajectory.F,
takes trajectory coordinate values and linearly interpolates in x,y, and z to get the
value of up 100 variables per trajectory per time step.  The code will detect and handle
trajectory coordinate values that are between the grid parcels.  The user may have
up to 1000 trajectories per domain and as mentioned up to 100 variables per trajectory.

Inputs are in the namelist files wrfinput_traj_d<domain>.
Outputs are in the netcdf files wrfout_traj_d<domain>.

LIST OF MODIFIED FILES:

M	Registry/Registry.EM_COMMON
M	Registry/Registry.NMM
M	chem/chem_driver.F
M	chem/depend.chem
M	dyn_em/module_em.F
M	dyn_em/start_em.F
M	external/RSL_LITE/module_dm.F
M	main/depend.common
M	share/Makefile
M	share/module_check_a_mundo.F
A	share/module_trajectory.F
M	share/solve_interface.F

TESTS CONDUCTED:

WTF reg test, version 3.05, has been successfully completed. The code has been tested in ACOM by
Mary Barth,  Megan Bella, and Gabrielle Pfister.
@stacywalters
Copy link
Contributor Author

stacywalters commented Jan 10, 2017 via email

@davegill
Copy link
Contributor

Stacy,
Looks like muus and muvs are "i1", not "state". So, get rid of the "grid%" part. The variables are just muus and muvs, not part of the grid DDT.
Dave

@stacywalters
Copy link
Contributor Author

stacywalters commented Jan 11, 2017 via email

KEYWORDS: trajectory

SOURCE: internal

DESCRIPTION OF CHANGES:

The original "trajectory" code in routine module_em.F only computed the trajectory
spatial coordinates per trajectory, per time step.  The new code, in share/module_trajectory.F,
takes trajectory coordinate values and linearly interpolates in x,y, and z to get the
value of up 100 variables per trajectory per time step.  The code will detect and handle
trajectory coordinate values that are between the grid parcels.  The user may have
up to 1000 trajectories per domain and as mentioned up to 100 variables per trajectory.

Inputs are in the namelist files wrfinput_traj_d<domain>.
Outputs are in the netcdf files wrfout_traj_d<domain>.

LIST OF MODIFIED FILES:

M	Registry/Registry.EM_COMMON
M	Registry/Registry.NMM
M	chem/chem_driver.F
M	chem/depend.chem
M	dyn_em/module_em.F
M	dyn_em/start_em.F
M	external/RSL_LITE/module_dm.F
M	main/depend.common
M	share/Makefile
M	share/module_check_a_mundo.F
A	share/module_trajectory.F
M	share/solve_interface.F

TESTS CONDUCTED:

WTF reg test, version 3.05, has been successfully completed. The code has been tested in ACOM by
Mary Barth,  Megan Bella, and Gabrielle Pfister.
Bring local TRJ branch up to latest master branch.
@jamiebresch
Copy link
Contributor

@davegill @stacywalters @mkavulich We will probably need to run a WRFDA regression test on the TRJ branch.

@stacywalters
Copy link
Contributor Author

stacywalters commented Jan 12, 2017 via email

@mkavulich
Copy link
Contributor

mkavulich commented Jan 12, 2017 via email

@stacywalters
Copy link
Contributor Author

stacywalters commented Jan 12, 2017 via email

@davegill
Copy link
Contributor

davegill commented Jan 12, 2017 via email

@jamiebresch
Copy link
Contributor

Yes, the TRJ branch works for all the DA tests Mike and I did.

Jamie

@mkavulich
Copy link
Contributor

mkavulich commented Jan 12, 2017 via email

@davegill davegill merged commit 8cb2192 into wrf-model:master Jan 12, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants