A few timing number

[cecilehannay]

I often get asked about timing and throughput. Here are a few numbers:

Coupled

L58 (B1850WscMOM + cice 5)
Model Cost: 6135.95 pe-hrs/simulated_year
Model Throughput: 8.73 simulated_years/day
Number of PE = 2232 (See pe_layout at #158)

Standalone

L58 (FWscHIST)
Model Cost: 5412.36 pe-hrs/simulated_year
Model Throughput: 9.58 simulated_years/day
Number of PE = 2160 (See pe_layout at #152)

L93 (FWscHIST)
Model Cost: 17931.19 pe-hrs/simulated_year
Model Throughput: 5.78 simulated_years/day
Number of PE = 4320 (See pe_layout at #40)

@JulioTBacmeister

[cecilehannay]

I added the total number of processors for each run, as it help to interpret the numbers.

[adamrher]

how much i/o is in these runs?

[adamrher]

we should run with ./xmlchange TIMER_LEVEL=10 then I can run my pie chart timing plot that will break it down in terms of i/o, physics, dynamics. Can we do this?

[adamrher]

ahh .. it's the time-stepping in L93:

se_nsplit=3
se_rsplit=6
se_hypervis_subcycle=3
se_hypervis_subcycle_sponge =  1

[PeterHjortLauritzen]

Just to add some more context:

The L58 model has a top around 42km and the time-steps are scaled for a max wind speed of 120m/s (CSLAM runs with a 900s time-step)
The L93 model has a top around 80km and the time-steps are scaled for a max wind of over 300m/s (CSLAM runs with a 600s time-step)

Hence tracer transport alone is about 33% more expensive due to the shorter time-step.

For WACCM with a 140km top the time-steps must be reduced further since the max winds in simulations is roughly 400m/s.

It is correct that cost scales linearly with number of levels if the model top remains the same. If the model top is moved up where large winds occur the scaling changes.

These results are independent of dycore. Here is an example of MPAS-WACCM (70 levels) showing maximum wind speed in the domain every dynamics time-step:

[Data courtesy of Soudeh Kamali, HAO, NCAR; the zero points are spurious - appears from restarting model]

The plot below (just for the interested reader) is the level/height at which the max/min winds occur: