bug: MPAS_ATM get_close_state puts all VERTISSURFACE state "far away" (dist=1.0e9_r8)

[hkershaw-brown]

🐛 Your bug may already be reported!
Please search on the issue tracker before creating a new issue.

Describe the bug

1. Run filter with some VERTISSURFACE state, e.g. t2m (QTY_2M_TEMPERATURE), QTY_10M_U_WIND_COMPONENT
   with vertical localization on, these surface fields in the state are not updated.
   with horizontal localization only, the surface fields in the state are updated.
2. What was the expected outcome?
   VERTISSURFACE fields updated if an observation is close (either with or without vertical location on).
3. If vertical location is on, none of the VERTISSURFACE fields are updated by the assimilation. The get_close_state distance is always 'far away' dist=1.0e9_r8

Error Message

No error, user reported that the t2m field was not being updated.

Which model(s) are you working with?

mpas_atm

There is a reproducer in /glade/derecho/scratch/hkershaw/DART/Bugs/mpas_2m_10m/small_run with a little mpas case (240km_uniform - will run single core on a laptop). This case is MPAS-8 recompiled to output reconstructed winds in the restart file to match the how the user was running. I believe this bug would present exactly the same if using DART with MPAS-7.

Version of DART

Which version of DART are you using?
v11.8.1
I believe this problem applies to all mpas-atm since at least Aug 14, 2020
591eecd (pull #44 730 files changed - scroll for mpas_atm)
This is when the convert_vert_distrib_state came in with regional mpas changes. I am not sure if the bug is present before that.

Have you modified the DART code?

No, the problem appears in the off-the-shelf code.

I changed istatus initialization in mpas_atm/model_mod.f90 convert_vert_distrib_state to 38 to confirm that the istatus is never changed:

DART/models/mpas_atm/model_mod.f90

Line 4880 in 464aa57

istatus = 1

get_close_state -> convert_vert_distrib_state -> fails conversion -> "far away"

There is no istatus setting for several of the case statements, so I think this is a deeper problem with the code that is used for state vertical conversion.

Note the incoming vertical gets hardcoded to VERTISLEVEL

DART/models/mpas_atm/model_mod.f90

Lines 4919 to 4921 in 464aa57

	zin(:) = vert_level
	ztypein = VERTISLEVEL
	zout(:) = missing_r8

Build information

Please describe:

1. Derecho
2. intel