CIME5 not duplicating CIME2 performance data collection functionality

[worleyph]

List of differences:

a) setting SAVE_TIMING to .true. is not renaming the timing subdirectory in the case run directory. It is still named "timing".

b) setting SAVE_TIMING to .true. is also not creating a file with the name "timing..saved" in the case timing subdirectory.

c) setting SAVE_TIMING to .true. and giving SAVE_TIMING_DIR a legal location is creating a gzipped tar file called "timing..tar.gz" in the performance archive in the correct place. However, gunzip; tar xf on this file generates, for example,

./lustre/atlas/scratch/worley/cli112/A_WCYCL2000.ne30_oEC_pgi_rljacob_cime5_d/run/timing

rather than "timing." as is currently done in CIME2.

d) TPROF_TOTAL is being set (by default) to 12 in env_run.xml. However, this is not setting tprof_n or tprof_option in drv_in to the corresponding values. These are currently

  tprof_n = -999
  tprof_option = 'never'
  tprof_ymd = -999

so no timing checkpoints are being generated.

e) The timing summary file, the one generated and put into the case timing subdirectory, is not also being copied to the performance archive. E.g. for CIME2 the performance archive directory for the run contains:

 cesm_timing.(case name).161205-135823
 cesm_timing_stats.161205-135823

while for CIME5 it contains only:

 acme_timing_stats.161205-173619

f) The checkpoint subdirectory in the performance archive is empty. I also see no information in the job output logs to indicate that the background monitoring job is ever launched. The Tools subdirectory does not have a script called mach_syslog either, so there seems to be no support for this functionality.

g) The SourceMods..tar.gz expands to, e.g.

  ./lustre/atlas1/cli112/proj-shared/performance_archive/worley/A_WCYCL2000.ne30_oEC_pgi_rljacob_cime5_d/161205-230030/autofs/nccs-svm1_home1/worley/ACPI/SVN/ACME/rljacob/cime/cime5-upgrade/ACME/cime/scripts/A_WCYCL2000.ne30_oEC_pgi_rljacob_cime5_d/SourceMods

instead of

  ./SourceMods

h) CaseStatus is missing information that was collected in CIME2. For comparison:

CIME2:

 create_newcase -case ACME_WCYCL2000.ne30_oEC_pgi_mastercheck_bnew -compset ACME_WCYCL2000 -res ne30_oEC -mach titan -compiler pgi -project cli112
 case created 2016-12-05 12:24:43
 <command>xmlchange BUILD_THREADED=FALSE</command>
 check input started 161205-123027
 build started 161205-123031
 <command>xmlchange SMP_BUILD=0,NINST_BUILD=0,BUILD_COMPLETE=FALSE,BUILD_STATUS=0</command>
 check input started 161205-124844
 build started 161205-124847
 build complete 161205-134530
 run started 2016-12-05 13:47:38
 job 3102067 created: ctime = Mon Dec 5 13:47:26 2016
 job 3102067 queued: qtime = Mon Dec 5 13:47:26 2016
 job 3102067 eligible to run: etime = Mon Dec 5 13:47:26 2016
 job 3102067 started: start_time = Mon Dec 5 13:47:33 2016
 model run started 2016-12-05 13:48:17
 model run ended 2016-12-05 13:54:12
 run SUCCESSFUL 2016-12-05 13:54:12
 CESM postrun ended 2016-12-05 13:54:15

CIME5:

  2016-12-05 13:52:53: case.setup starting
  2016-12-05 13:53:28: case.setup complete
  2016-12-05 13:58:05: case.build starting
  2016-12-05 14:16:00: case.build complete
  2016-12-05 17:37:03: Run started
  2016-12-05 17:55:17: Run SUCCESSFUL

I realize that the create_newcase command is now in README.case, and that this is now in the CaseDocs. subdirectory, so this is fine.

Some of the other information is system specific (batch system specific) information on job submission and scheduling times. This is very useful in understanding end-to-end throughput rates.

Finally information is missing on the cost of the pre- and postprocessing. As this typically counts against the allocation, even though not run on the compute nodes, it can be important to monitor this.

i) In the case timing subdirectory CIME5 is compressing the performance summary file and not compressing the raw global statistics timing data. In CIME2 the opposite was true (and makes more sense). The global statistics file can be long, and the summary file is more likely to be read in any case (or, at least, read first).

[jgfouca]

@worleyph My PR address most of your concerns:

a) Fixed
b) Fixed
c) Fixed
d) This is a bug but it's been fixed in CIME-5.2 which significantly refactored namelist building, so I'm going to request that you wait until CIME-5.2 is merged into ACME.
e) Fixed
f) Fixed (I think, mach_syslog is definitely restored, but checkpoints are empty due to (d) so it's hard to test)
g) Fixed
h) Should be fixed with CIME-5.2
i) Fixed

[worleyph]

@jgfouca , Thanks! I'll give it a try. Note that (f) can be tested by setting tprof_n and tprof_option in user_nl_cpl. Also, this just determines how many timing file checkpoints there are. If the jobs runs longer than SYSLOG_N seconds, other files will also be copied into the checkpoint subdirectory. I'll try this as well.

[worleyph]

Hi @rljacob , we have a number of issues that are closed because they are fixed in CIME5.2 . Personally, I'd prefer that they not be closed until the issues are fixed in ACME, if nothing else so that it will remind us (or me) to test that the fixes work as expected in ACME.

[rljacob]

Yes I agree. Its not closed in ACME until the fix is on master in ACME. Feel free to re-open any still broken issues.

[worleyph]

Reopening until fixes for (d) and (h) are in place and tested in ACME.

[jgfouca]

This is now on master.

[worleyph]

Please let me test first.

[worleyph]

Closing this and creating a new issue if needed - (d) is now resolved, but (h) is different - will evaluate.