Enable physics load balancing for CAM-SE by default

[worleyph]

Recent ACME experiments across all target platforms, mesh sizes,
and process and thread counts indicate that performance
is improved (or, at least, not degraded) by using physics load
balancing option 2 with the point-to-point implementation of
the load balancing communication operator (phys_alltoall == 1).
This changes the defaults to these option settings. Other options,
including the original defaults, can be set in user_nl_cam.

This is bit-for-bit compared to not using physics load balancing
if the BFBFLAG in env_run.xml is TRUE, which is the current ACME
default.

[BFB]

[worleyph]

I did a sanity check for this change - it did change the defaults. I have been testing this / using this pretty extensively over the past few months, setting the options manually in user_nl_cam.

With BFBFLAG now TRUE by default, this should be BFB, and easy to integrate?

I'm willing to run acme_developer on this, but Titan is unusable at the moment (Gordon Bell runs???), and I'd also like to reference existing baselines. Any suggestions @singhbalwinder ?

Note @golaz and @PeterCaldwell , you could get a performance bump from this.

[worleyph]

@golaz and @PeterCaldwell , you could also see performance issues I suppose, given Edison's poor history in consistent MPI performance. I'm hoping for the best. In recent A_WCYCL2000S ne30_oECv3 experiments on Edison (February) it increased ATM performance by 4 SYPD in the 375 node layout, and by 1 SYPD for the total run (1 month, no restart), from 9.93 to 10.96 . Since this looks nothing like the performance of your recent runs (lack of high frequency I/O?) I don't know what you will observe.

[singhbalwinder]

@worleyph : Running acme_developer will ensure that the model is running fine in various configurations. Given the very minor changes in this PR, I doubt we will see any issues. Is it possible for you to run acme_developer on EOS? If not, I can run those on this branch. For BFB comparisons, I generally choose just one test (e.g. FC5AV1C-L with ne4 grid) as it take a lot of time to generate and compare against baselines for all the tests in acme_developer.

[worleyph]

@singhbalwinder , can you please run on EOS? I haven't run on EOS for a long time. I'll do a BFB test on Titan. Note that while the code change is very small, it directs the code to go through different logic and to use a completely different decomposition in the atmospheric physics, so is not really a "small" change. Physics load balancing is not a new feature however, and hasn't changed for a long time.

[singhbalwinder]

@worleyph : I will run acme_developer on EOS.

[worleyph]

Using

  -compset FC5AV1C-L -res ne4_ne4 -mach titan -compiler pgi 

and both 16x1 and 16x4 decompositions, determined that atm.log files were B4B with this PR and unmodified master, i.e. comparing phys_loadbalance == 2 and phys_alltoall == 1 with phys_loadbalance == 0 . The PR runs were also faster.

[singhbalwinder]

Thanks @worleyph !

[singhbalwinder]

I ran the acme_developer tests on EOS and they all passed except the expected fails.

[singhbalwinder]

@mt5555 : If you are okay with these modifications, I will merge these to next today.

[rljacob]

Don't merge these to next. A non-BFB change is going on.

[rljacob]

As expected, this is creating a namelist diff in all F-cases. But shouldn't it also show up in fully coupled cases? It isn't for some reason.

[worleyph]

Sorry - I spaced that this would changed the namelists. I changed the default for phys_alltoall directly in the code, but there was already CAM namelist logic (a switch depending on whether se or eul dynamics were used) and phys_loadbalance was already in the atm_in namelist.

I just checked with an A_WCYCL compset - phys_loadbalance is being set to 2, so there should have been a namelist difference.

[worleyph]

@singhbalwinder , what is the status of this? Thanks.

[singhbalwinder]

I am going to merge it to master now and I will also add NML tag to this PR description.

[worleyph]

Thank you. I lost track of what was happening.