Performance Data Capture and Provenance in CIME5.2 (discussion)

[worleyph]

CIME 5.2 has lost a few minor capabilities / does things a little differently than before.

a) software_environment.txt is created by case.setup, not case.build . if env_mach_specific is changed, or it the system defaults change, between setup and build, this will not be captured.

b) CaseStatus looks different (note @bmayerornl ). Some of this is an improvement - job submission information is platform-agnostic, for example. Time spent in the pre-run and post-run is missing however, and this is probably important to track, especially if we are trying to minimize this further.

c) Compile time per component is no longer output. Measurement of this is somewhat problematic now as all components are built concurrently, but which one takes the longest might still be useful information.

d) I'll be updating the mach_syslog routines to look more like the optimizations that @amametjanov suggested (and were implemented) for Anvil. Might also eliminate some data that seemed like it might be useful, but which we never look at. ( @bmayerornl , you should weigh in here as well.)

[rljacob]

I had also complained about the compile time per component. I believe the splitting by component is done at the python level so it should be possible to time each thread (?).

My complaint about CaseStatus was all the extra newlines. That was a side effect from making TestStatus more readable.

[worleyph]

With PR #1429 the performance data provenance being collected looks like:

a) Case information:

 CaseDocs.$lid/

(CaseDocs from the case directory as well as all env_xxx files, user_nl_xxx files, case.run, Depends file, README.case, and software_environment.txt)

b) Code information:

 GIT_DESCRIBE.$lid
 SourceMods.$lid.tar

c) Environment information (build and run):

 build_environment.txt.$lid
 run_environment.txt.$lid

d) "What happened" information, including runtime per day (cpl.log) and actual PIO strides, etc. (cpl.log and acme.log)

 CaseStatus.$lid
 $jobid.OU.$lid (titan and anvil), $case.$jobid.$lid.gz (SLURM), $jobid.output.$lid and $jobid.cobaltlog.$lid (cetus and mira)
 acme.log.$lid
 cpl.log.$lid

e) Performance data:

 acme_timing.$case.$lid
 acme_timing_stats.$lid
 timing.$lid.tar

f) System status when job started (system-specific - see provenance.py for actual commands):

 Titan: qstatf_jobid.$lid, showq.$lid, xtnodestat.$lid
 SLURM: sqsf_jobid.$lid, squeuef.$lid, squeues.$lid, sinfol.$lid
 Mira/Cetus: qstatf.$lid, qstatf_jobid.$lid
 Anvil: qstatf.$lid, qstatf_jobid.$lid, qstatr.$lid

g) Then the checkpoints subdirectory captures snapshots of progress during the run (at SYSLOG_N second intervals, where frequency of checkpoint timing data is controlled by TPROF_N):

g1) Progress for each component (in terms of component-specific metrics, usually something like step) appended to the following files every SYSLOG_N seconds

 atm.log.$lid.step
 cpl.log.$lid.step
 ice.log.$lid.step
 lnd.log.$lid.step
 ocn.log.$lid.step
 rof.log.$lid.step

g2) All runtimes per day from cpl.log up to that point

  cpl.log.$lid.step-all

g3) System status information (what other jobs are running, and where). $remaining suffix is how much time is left before job exceeds requested wallclock time.

 titan: showqr.$lid.$remaining, xtnodestat.$lid.$remaining
 SLURM: squeuef.$lid.$remaining, squeues.$lid.$remaining
 mira/cetus: qstatf.$lid.$remaining
 anvil: qstatr.$lid.$remaining, qstatn.$lid.$remaining

g4) acme.log right after job has started (on Titan, makes sure that get process and thread to core mapping information that shows up first)

 acme.log.$lid.$remaining

g5) checkpoint timing data for root PE for each component, and also global statistics. Copied over at SYSLOG_N intervals, but generated at TPROF_N-derived frequency, e.g. for every day with all components have process 0 as root:

 model_timing_$simulationtime.$processid
 model_timing_$simulationtime_stats

[jgfouca]

@worleyph I'm looking at this ticket. Is there anything you want me to do here? I could potentially handle (b) and (c) from your list.

[worleyph]

@jgfouca , this is just a discussion thread so far. Once we decide on something, if we do, we'll generate a new github issue. Thanks for asking though.

[rljacob]

@worleyph should this still be open?

[worleyph]

Probably not. If okay with you, I'll wait for the new CIME to be integrated and then check whether any items in the original list are missing and still might be useful. I think that most have been addressed. At this point I'll close it and open a new issue if necessary.

[jgfouca]

@worleyph can I close?

[worleyph]

I'll take care of this by the end of the week. I'll definitely close it, but will first check if there is anything that I want to move to a new issue. Thanks for the reminder though.

[worleyph]

Closing this now. Created github issue #1857 for item (b) and will start working on a PR to address this.