Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running with GEOSchemchem uses high amounts of memory #19

Open
kgerheiser opened this issue Oct 7, 2019 · 2 comments
Open

Running with GEOSchemchem uses high amounts of memory #19

kgerheiser opened this issue Oct 7, 2019 · 2 comments
Labels
bug Something isn't working

Comments

@kgerheiser
Copy link

kgerheiser commented Oct 7, 2019

Running on 6 processors and a single node at C48 I am seeing ~18 GB per process, and if emissions are turned on it won't even run. That, coupled with the slow memory leak, means I can only run for several hours before it crashes.

Upping the number of nodes and the number of processes does not help much, and the consumed memory remains constant at around 80% from 1 node and 6 processes or 10 nodes and 280 processes.

Considering this chemistry can run within the AGCM it doesn't seem right that the CTM should consume a whole 128 GB node worth of memory at C48.

I am currently investigating with TotalView's memory checker.

@kgerheiser kgerheiser added the bug Something isn't working label Oct 7, 2019
@tclune
Copy link
Collaborator

tclune commented Oct 7, 2019

This sounds like a known issue. MAPL develop is much better with regard to leaking memory. (Actually it is the newer version of ESMF that improves it).
But if you are seeing 18 GB/proc at the beginning of the run, there is a problem.

@kgerheiser
Copy link
Author

Yes, it's using 18 GB at the start of the run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants