Add validation of max temp. during dynamic adjustment

[xylar]

This will hopefully catch issues like #308

[xylar]

Testing

I'm testing dynamic adjustment on Anvil and Intel with this test.

• EC30to60: this fails as expected

ValueError: Max of temperatureMax > allowed threshold: 54.22333235731241 > 33.0 in 
/lcrc/group/e3sm/ac.xylar/compass_1.0/anvil/test_20220321/dyn_adj_ec_validate_temp_redi_min_6/
ocean/global_ocean/EC30to60/PHC/dynamic_adjustment/damped_adjustment_2/analysis_members/
globalStats.0001-01-11_00.00.00.nc

• ECwISC30to60: this fails as (more or less) expected

ValueError: Max of temperatureMax > allowed threshold: 35.95204635559652 > 33.0 in 
/lcrc/group/e3sm/ac.xylar/compass_1.0/anvil/test_20220321/dyn_adj_ecwisc_validate_temp_redi_min_6/
ocean/global_ocean/ECwISC30to60/PHC/dynamic_adjustment/damped_adjustment_2/analysis_members/
globalStats.0001-01-11_00.00.00.nc

• QU240
• QUwISC240
• SOwISC12to60: Failing in ssh_adjustment (needs Add bathymetry smoothing to global ocean init #310 before it's going to work)
• WC14

[xylar]

I need to make sure this threshold is reasonable.

[xylar]

@mark-petersen, this is ready for your review. Is 33.0 C a good threshold in you opinion?

[mark-petersen]

@xylar this is a great idea! Thanks for implementing it. Yes, I think 33C is an appropriate value. We can be conservative here, as it will tell us if the max is increasing at all.

Approving based on this catching increasing temperatures in the tests above.

[xylar]

Thanks again @mark-petersen!