Problematic values for dynamics tendencies of the x and y (u and v) winds on restart

[gspetro-NOAA]

Discussed in #2141

^{Originally posted by ericaligo-NOAA February 20, 2024}
Following the instructions here: https://ccpp-techdoc.readthedocs.io/en/latest/ParamSpecificOutput.html
I was able to output and plot various physics and dynamics tendencies. I noticed that the x and y wind tendencies from the dynamics have reasonable values until the first output time after my restart. I see values such as Contouring: -555556 to -555556 interval 0.01. I have hourly output until my restart time when I have output every 3 minutes. My fhzero is set to 3 hours. All other tendencies seem okay. Is this a bug?

There is an ufs-community/ccpp-physics#175 to look into this problem, but
@dustinswales and @grantfirl were thinking the issue is likely somewhere above the physics, possibly related to restart, so someone on the WM side should likely look into it.

@grantfirl observed that:

• The x and y wind dynamics tendencies are "non-physics" tendencies calculated between the timestep_init and run phase of the radiation group and are not touched by the CCPP at all.
• These variables are part of the Diag DDT in GFS_typedefs and are not needed for restart, so the problem may have something to do with the values of statein%ugrs/vgrs and stateout%gu0/gv0 for the first time update_atmos_radiation_physics is called after a restart.
• Since the array "dtend" holding the x and y wind dynamics tendencies seems to be properly initialized to zero before the first call, it seems that one of ugrs/vgrs or ug0/vg0 is not set to something meaningful yet.
• It is odd that other non-physics tendencies (e.g., T, qv, and o3) seem fine bc they seem to be handled identically.