Merge remote-tracking branch 'origin/worleyph/cam/twin_alg_init_opt' PR

#1347

Optimize physics load balancing twin algorithm initialization

When load balancing physics computation on a latlon mesh,
there is an advantage to assigning columns to chunks in pairs,
pairing each column with its 'twin' in the other hemisphere,
latitude reflected around the equator, and offset 180 degrees
in longitude. While a similar approach should be advantageous,
or at least no worse than no doing it, with a non-latlon mesh,
the cost of initialization of this 'twin' algorithm was too high
for the HOMME spectral element mesh to be acceptable even though
only part of the initialization cost. Because of this, the twin
algorithm is disabled by default when using a non-latlon mesh.
(For a latlon mesh, the twin algorithm is enabled by default.)
Note that a runtime parameter can be used to override the default.

A simple change to the initialization algortihm drops the cost
to an acceptable level, less than 2 minutes even on one core of
Mira and for an ne120 HOMME mesh. In performance experiments, using
the twin algorithm has not improved the load balance compared to
not using it for current ACME production-like cases, and the default
disabling the twin algorithm is not being changed. However, this
optimization to the initialization algorithm will make it feasible
to re-examine this option in the future.

This change will not be tested by the ACME test suites, as the
twin algorithm is disabled, and it is also irrelevant unless
atmospheric physics load balancing is enabled.

Experiments with F compsets and the CAM-SE dycore for both
ne30 and ne120 meshes have been conducted, indicating that the
same load balancing redistribution of columns is computed by
the twin algorithm with and without this optimization. Tests with
FV and Spectral Eulerian dycores have not been conducted.

[BFB]