Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support for Anvil in provenance.py was dropped #1417

Closed
worleyph opened this issue Apr 15, 2017 · 4 comments · Fixed by #1429
Closed

support for Anvil in provenance.py was dropped #1417

worleyph opened this issue Apr 15, 2017 · 4 comments · Fixed by #1429
Assignees
@jayeshkrishna
Labels
Anvil bug CIME

Comments

@worleyph
Copy link
Contributor

PR #1248 added support for calling mach_syslog and collecting other provenance-related data for Anvil to provenance.py. This has since disappeared, and it is not in CIME 5.3 either. I would appreciate that this be restored. I can submit another PR if required but would like this to go in quickly.
@rljacob rljacob assigned jgfouca and jayeshkrishna and unassigned rljacob Apr 16, 2017
@worleyph
Copy link
Contributor Author

I have branch that I am putting together for synchronizing mach_syslog logic across all of the systems. I'll put the Anvil support back in in this and then submit a PR. I would like to understand how Anvil support got dropped though. I also want this new PR to go into ACME directly, and not wait for it to go through CIME. Most of the changes will be in (what was) cime_config/acme/machines, but provenance.py will need to be changed as well.

@jgfouca
Copy link
Member

jgfouca commented Apr 16, 2017

@worleyph probably got messed up in the ESMCI/ACME integration. I think the likelihood of these kinds of mistakes will decrease now that we are doing more frequent integrations. Sorry for the screwup.

@worleyph
Copy link
Contributor Author

Other than the increased frequency, will the process be any different? Does not seem to be a frequency related issue.

I assume that I need to wait for the CIME 5.3 integration to make it to master, and to start over, since everything has moved around? Thanks.

@jgfouca
Copy link
Member

jgfouca commented Apr 16, 2017

@worleyph , increased frequency should lead to greatly reduced conflicts when we do merges and resolving conflicts is where mistakes can be made.

Yes, it's probably wise to hold off on CIME changes until 5.3 merge is complete. It's already on next and should be on master soon.

@worleyph worleyph mentioned this issue Apr 20, 2017
Merged
@jgfouca jgfouca removed their assignment Apr 20, 2017
amametjanov added a commit that referenced this issue Apr 23, 2017
@amametjanov
Merge branch 'worleyph/cime/update_provenance_collection' into next (PR
ddf2869 
#1429)

Update provenance data collection:
a) Update mach_syslog for other platforms to match that for Anvil

When adding mach_syslog for Anvil, the logic was refactored,
decreasing the amount of data collected. Introduce similar changes
for mach_syslog for the other systems. The system-specific
commands are querying system and job status were also examined,
and some of the command choices and options modified.

b) Modify collection of provenance data on all supported systems

Disable collection of xtdb2proc and mdiag system configuration
data and qstat -f system load data or Titan. (May want to re-enable
in the future, so just commenting out, not deleting.)

Eliminate collection of some redundant or irrelevant date on SLURM
systems at NERSC, and add collection of some data on system
configuration and node usage by other jobs. Also fix error caused
by changed name of ACME job script output on NERSC systems.

Anvil support in provenance.py was lost inadvertently. Add it
back. Also eliminate some unnecessary detail in the provenance data
collected on Anvil.

Eliminate some unnecessary detail in the provenance data collected
on Mira and Cetus.

[BFB]

Fixes #1417
amametjanov added a commit that referenced this issue Apr 27, 2017
@amametjanov
Merge branch 'worleyph/cime/update_provenance_collection' (PR #1429)
68ba6d3 
Update provenance data collection:
a) Update mach_syslog for other platforms to match that for Anvil

When adding mach_syslog for Anvil, the logic was refactored,
decreasing the amount of data collected. Introduce similar changes
for mach_syslog for the other systems. The system-specific
commands are querying system and job status were also examined,
and some of the command choices and options modified.

b) Modify collection of provenance data on all supported systems

Disable collection of xtdb2proc and mdiag system configuration
data and qstat -f system load data or Titan. (May want to re-enable
in the future, so just commenting out, not deleting.)

Eliminate collection of some redundant or irrelevant date on SLURM
systems at NERSC, and add collection of some data on system
configuration and node usage by other jobs. Also fix error caused
by changed name of ACME job script output on NERSC systems.

Anvil support in provenance.py was lost inadvertently. Add it
back. Also eliminate some unnecessary detail in the provenance data
collected on Anvil.

Eliminate some unnecessary detail in the provenance data collected
on Mira and Cetus.

[BFB]

Fixes #1417
jgfouca pushed a commit that referenced this issue Jun 2, 2017
@amametjanov
Merge branch 'worleyph/cime/update_provenance_collection' (PR #1429)
c37feff 
Update provenance data collection:
a) Update mach_syslog for other platforms to match that for Anvil

When adding mach_syslog for Anvil, the logic was refactored,
decreasing the amount of data collected. Introduce similar changes
for mach_syslog for the other systems. The system-specific
commands are querying system and job status were also examined,
and some of the command choices and options modified.

b) Modify collection of provenance data on all supported systems

Disable collection of xtdb2proc and mdiag system configuration
data and qstat -f system load data or Titan. (May want to re-enable
in the future, so just commenting out, not deleting.)

Eliminate collection of some redundant or irrelevant date on SLURM
systems at NERSC, and add collection of some data on system
configuration and node usage by other jobs. Also fix error caused
by changed name of ACME job script output on NERSC systems.

Anvil support in provenance.py was lost inadvertently. Add it
back. Also eliminate some unnecessary detail in the provenance data
collected on Anvil.

Eliminate some unnecessary detail in the provenance data collected
on Mira and Cetus.

[BFB]

Fixes #1417
jgfouca pushed a commit that referenced this issue Jun 2, 2017
@jedwards4b
Merge pull request #1597 from ESMCI/jgfouca/infer_machine_from_tests
7003fe2 
Add ability for create_test to infer non-default machine from tests
Test suite: scripts_regression_tests --fast
Test baseline:
Test namelist changes:
Test status: bit for bit

Fixes #1417

User interface changes?: None

Code review: @jedwards4b
jgfouca pushed a commit that referenced this issue Feb 27, 2018
@amametjanov
Merge branch 'worleyph/cime/update_provenance_collection' (PR #1429)
8503285 
Update provenance data collection:
a) Update mach_syslog for other platforms to match that for Anvil

When adding mach_syslog for Anvil, the logic was refactored,
decreasing the amount of data collected. Introduce similar changes
for mach_syslog for the other systems. The system-specific
commands are querying system and job status were also examined,
and some of the command choices and options modified.

b) Modify collection of provenance data on all supported systems

Disable collection of xtdb2proc and mdiag system configuration
data and qstat -f system load data or Titan. (May want to re-enable
in the future, so just commenting out, not deleting.)

Eliminate collection of some redundant or irrelevant date on SLURM
systems at NERSC, and add collection of some data on system
configuration and node usage by other jobs. Also fix error caused
by changed name of ACME job script output on NERSC systems.

Anvil support in provenance.py was lost inadvertently. Add it
back. Also eliminate some unnecessary detail in the provenance data
collected on Anvil.

Eliminate some unnecessary detail in the provenance data collected
on Mira and Cetus.

[BFB]

Fixes #1417
jgfouca pushed a commit that referenced this issue Mar 14, 2018
@amametjanov
Merge branch 'worleyph/cime/update_provenance_collection' (PR #1429)
b5c009e 
Update provenance data collection:
a) Update mach_syslog for other platforms to match that for Anvil

When adding mach_syslog for Anvil, the logic was refactored,
decreasing the amount of data collected. Introduce similar changes
for mach_syslog for the other systems. The system-specific
commands are querying system and job status were also examined,
and some of the command choices and options modified.

b) Modify collection of provenance data on all supported systems

Disable collection of xtdb2proc and mdiag system configuration
data and qstat -f system load data or Titan. (May want to re-enable
in the future, so just commenting out, not deleting.)

Eliminate collection of some redundant or irrelevant date on SLURM
systems at NERSC, and add collection of some data on system
configuration and node usage by other jobs. Also fix error caused
by changed name of ACME job script output on NERSC systems.

Anvil support in provenance.py was lost inadvertently. Add it
back. Also eliminate some unnecessary detail in the provenance data
collected on Anvil.

Eliminate some unnecessary detail in the provenance data collected
on Mira and Cetus.

[BFB]

Fixes #1417
mark-petersen pushed a commit to mark-petersen/E3SM that referenced this issue Jan 19, 2021
@matthewhoffman
Merge branch 'develop' into landice/develop
65b5f28 
Update needed for ACME

Framework features brought in:

* f444d0f Merge PR E3SM-Project#1418 'matthewhoffman/framework/output_record_reference_time' into develop
* 263e14f Merge PR E3SM-Project#1428 'mark-petersen/framework/couple_fixes' into develop
* bcce31d Merge PR E3SM-Project#1424 'amametjanov:az/tools/cp-prebuilt-tools' into develop
* 98cfeea Merge PR# 1349 'akturner/framework/forcing_cleanup' into develop
* 9359319 Merge PR E3SM-Project#1347 'akturner/framework/forcing_restart_timestamp' into develop
* e9ce203 Merge PR E3SM-Project#1348 'akturner/framework/forcing_at_init' into develop
* 4974284 Merge PR E3SM-Project#1368 'akturner/framework/improved_messages_in_driver' into develop
* 86d50c5 Merge PR E3SM-Project#1417 'akturner/framework/forcing_multiple_blocks' into develop
* 9116da3 Merge branch 'framework/validation-of-streams-using-interval_in-interval_out' into develop
* e466b46 Merge branch 'framework/interval_in-interval_out-support-for-streams' into develop
* 30dc955 Merge branch 'az/framework/mpas_dmpar-race-fix' into develop
* b632938 Merge branch 'framework/i8_interval_division' into develop
* 6dac06c Merge branch 'framework/log_write_IBM_error' into develop
* 960a648 Merge branch 'framework/cleanup-logging-stream-manager' into develop
* 504c282 Merge branch 'framework/make-streams-with-direction-none-inactive' into develop
* 5903748 Merge branch 'framework/correctly_remove_blk_fields' into develop
* 3565965 Merge branch 'framework/iostreams-real4dfield-bug' into develop
* 8b60591 Merge branch 'framework/missing-deallocate-nEdgesOnCellField-bootstrapping' into develop
* 70b953b Merge branch 'master' into develop
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jayeshkrishna jayeshkrishna

Labels
Anvil bug CIME
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update provenance data collection E3SM-Project/E3SM
4 participants
@rljacob @jgfouca @worleyph @jayeshkrishna