Move case bld directory to $HOME/acme_scratch/$PROJECT/$CASE on titan

[jgfouca]

Explicitly set titan's CESMSCRATCHROOT to a subdirectory within
the user's home and ensure rundir gets set to a place that the
compute nodes can see.

[BFB]

[rljacob]

@worleyph can you take care of this this week? Its needed for the automatic testing on Titan.

[worleyph]

Can someone describe (again) what the original problem was, and why this solves this? Nothing is changing except moving the bld directory out of the case directory and putting it in a new acme_scratch directory in the users home directory. I don't like this personally, having gotten used to the existing location (in the case directory). I have run the test suites with the current setup (though not recently), with no problems.

[jgfouca]

@worleyph , if your build directory in is /scratch, you get disastrously slow build times, if your run directory is not in /scratch, then the compute nodes cannot see it. This change forces these directories to go into the right places.

This change also simplifies our nightly testing situation. Our jenkins scripts use $CESMSCRATCHROOT/jenkins as the testroot for create_test. I'd like to not have to make a special case for titan.

Looking at it differently, this change bring titan's behavior in-line with all our other machines. No other machine puts its build directories under CASEROOT.

[worleyph]

if your build directory in is /scratch, you get disastrously slow build times

agreed

if your run directory is not in /scratch, then the compute nodes cannot see it.

agreed

This change forces these directories to go into the right places.

The current settings also have the required behavior.

Our jenkins scripts use $CESMSCRATCHROOT/jenkins as the testroot for create_test.

Okay - this is the basis of the problem then.

No other machine puts its build directories under CASEROOT.

No other system builds in the home directory. This changes the meaning of $CESMSCRATCHROOT in an equally unexpected way, in that scratch is not the "usual" scratch location. So Jenkins requirements are screwing around with other ways that the model is built and used, which is also different from other systems.

I'll give this some thought this afternoon.

[worleyph]

Okay ... if we are going this direction, I'd like to make one change to the proposal:

$ENV{HOME}/acme_scratch/$PROJECT

(adding $PROJECT to the definition of CESMSCRATCHROOT; this reintroduces some symmetry with the $RUNDIR location that was lost previously).

This still takes some control away from the user - I have not given up on building in lustre. It used to work, and may again some day, and may be a requirement in some cases when the home directory is too full to permit builds there. The workaround will require the user to modify env_build.xml, so this is just an educational issue.

Please push my suggestion to your pull request. I'll then do a quick test. I assume that I can skip waiting for the automatic testing in next? I haven't done the "quick and dirty" merging process before. Can I ask @mrnorman to do this? Matt should weigh in on this anyway, given that he is the Titan POC.

Thanks.

[jgfouca]

@worleyph , as to your previous comment, I agree with most of what you said. I will just note that the correctness of the current settings depend on the user's selection of CASEROOT. The problem I was seeing in Jenkins is that it was picking a caseroot based on the CESMSCRATCHROOT and therefore builds took forever.

I do agree that CESMSCRATCHROOT is not typically a home directory. Melvin is the only other machine where this is the case. We got forced into this situation by the terrible performance of titan's scratch area.

I am fine with adding PROJECT in the way you suggested.

[jgfouca]

I just force pushed the $PROJECT change.

There's no need to wait for automatic testing, it isn't working on Titan anyway (times out). You can give this PR to @mrnorman if you'd like.

[jgfouca]

@worleyph , I should add that I'm open to keeping the build directory as a derivative of CASEROOT. The only thing that really must change in my opinion is the CESMSCRATCHROOT.

[worleyph]

Probably irrelevant, but I just ran three experiments on Titan, all

 ./create_newcase  -res ne30_m120 -compset A_B1850 -mach titan -compiler pgi -project cli112 ...

on titan-ext8 with a load of around .53 when not building (this was true between each experiment):

  13:50:56 up 31 days,  3:55, 38 users,  load average: 0.53, 0.52, 1.05
 USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
 worley   pts/3     12:48    0.00s  0.84s  0.02s w
 root     pts/0     09Nov15 21:22m  0.32s  0.18s -bash
 root     pts/1     02Nov15 29days  0.29s  0.18s -bash

a) source and build directory in $HOME.

 Compile time: 40m 03s

b) source and build directory in /lustre/atlas1/cli112/scratch

 Compile time: 51m 01s

c) source in home and build directory in /lustre/atlas1/cli112/scratch

 Compile time: 50m 57s

So penalty (when building all components) of building in /lustre/atlas1/scratch/cli112 is approximately 25% at the moment.

(Note, atmosphere build is between 13-15m. land is 7-9m, ice is 6-7m, ocn is 8-9m .)

[jgfouca]

25% isn't bad. The nightly process uses the ACME create_test, so a large number of builds are kicked off at once. What impact would that have on the difference between scratch and home?

[worleyph]

What impact would that have on the difference between scratch and home?

I have no idea. Perhaps lustre would not handle this as well as home, but ... we would need to determine this experimentally.

[jgfouca]

Let me do some timings

[worleyph]

Just for fun, I changed the optimization flags from -O2 to -O0 for the source+build in $HOME. This decreased compile time to 28m 36s . I'll try the lustre builds with this as well when I get the chance,

[worleyph]

@jgfouca , does jenkins clean up after itself? Note that

XXX.clean_build all

does not delete old executables, or the source copied over from ocn and ice, or weird stuff like:

./pgi/mpich/nodebug/threads/MCT/noesmf/clm/obj

(@rljacob, it looks like the clean_build logic doesn't do everything that it should for cime?)

This will quickly fill up a home directory allocation if the bld directories are not simply deleted.

[jgfouca]

@worleyph , yes, Jenkins attempts to remove all artifacts from the previous run.

[worleyph]

@jgfouca , things are not going smoothly. When testing your branch, and then when testing next after merging in your branch, the build is dying with the error message:

PGF90-S-0285-Source line too long (/autofs/nccs-svm1_home1/worley/ACPI/SVN/ACME/jgfouca/machines-acme/change_titan_scratch/ACME/components/homme/src/share/prim_advection_mod.F90: 1647)

This line is a comment, and is identical to the version in master as fas as I can tell. master builds fine with both the original location of ./bld and with your new location, introduced by editing config_machines.xml manually.

I need you to figure this out, perhaps by creating a new branch from master and adding your commit again. If you can then verify that a

-res ne30_m120 -compset A_B1850

case works for you, I'll look at merging this in again.

[jgfouca]

@worleyph , OK I'll look at it.

[jgfouca]

@worleyph , I'm not seeing the build error. Probably because my repo is not living in such a long path. The comment should probably be changed to not be so fragile, but that's an unrelated issue to this PR.

[worleyph]

@mrnorman , could you please take this over? Verify that it works for you, and then do the usual merge without further testing. This is Titan, can compile-only, specific, so any successful build should be a sufficient test. Thanks.

[jgfouca]

@worleyph , I'm getting timings now. It looks like create_test does not run that much faster in HOME than on scratch. It may not be worth the trouble.

[worleyph]

Hard decision ... the poor performance in the past was very real, and it could return. My suggestion is to put this on hold for the moment, but if you see slow build times return (and can attribute it to the file system), then complete the pull request. However, @mrnorman should weigh in here as I would have no qualms about this PR being merged in.

[jgfouca]

@worleyph , sorry to keep changing my tune, but all the data finally came in:
HOME: create_test took 5188.50154686 seconds
SCRATCH: create_test took 9238.61350107 seconds
This was for "./create_test --no-run acme_developer". So the time savings are significant.

[worleyph]

Great - this is unambiguous then.

[jgfouca]

@mrnorman , thoughts?

[mrnorman]

Sounds like a good change to me. Am I still on hold? I remember Pat asking me to hold off while something was fixed.

[jgfouca]

No, you are free to merge.

[worleyph]

@mrnorman , since the "hold" request I reassigned this to you. My only request now is that you verify that this works for you (single build for any case) before the merge.

[mrnorman]

Something has come up, and I have to take today and tomorrow off from this point. i'll work on this on Monday.

[worleyph]

If I can get to this, I'll give it a try. If not, it will still be yours on Monday :-).

[worleyph]

Passed a simple build test. SInce this is Titan and build specific, skipping acme-developer (before next) and integration (before master).

@jgfouca , is there a GitHub issue # or a Jira task associated with this?

[rljacob]

SEG-143

[worleyph]

Also, @jgfouca , how did you test this? Did you run the full acme_developer test suite or integration test suite in your performance evaluations, or just a single case?
(A single case is all that should be necessary, but might as well record what was actually done.)

[worleyph]

Integration testing skipped.

[worleyph]

@jgfouca , you should advertise to ACME the impact of this change, so people do not get confused about where things are now built on Titan, and where they need to go and what they need to do if their home directories fill up.

[worleyph]

@jgfouca , should I delete this branch? (Or do you want to do this?)

[worleyph]

@jgfouca - I'll make the announcement.

[jgfouca]

@worleyph, I ran "./create_test --no-run acme_developer"

[jgfouca]

I will delete, thanks for integrating.