Fatal PGI compiler bug when building B case with CLM4.5 on Titan

[worleyph]

for
./create_newcase -case ne30_B1850C5L45BGC_pgi -mach titan -compiler pgi -compset B1850C5L45BGC -res ne30_g16

builds with the PGI compiler are dying with

pgf90-Fatal-/opt/pgi/14.7.0/linux86-64/14.7/bin/pgf901 TERMINATED by signal 11
gmake: *** [ActiveLayerMod.o] Error 127

and

pgf90-Fatal-/opt/pgi/14.7.0/linux86-64/14.7/bin/pgf901 TERMINATED by signal 11
gmake: *** [dynCNDVMod.o] Error 127

and

pgf90-Fatal-/opt/pgi/14.7.0/linux86-64/14.7/bin/pgf901 TERMINATED by signal 11
gmake: *** [lnd2glcMod.o] Error 127

The same ActiveLayer error occured whether using pgi/14.2.0 or pgi/14.7.0 . I tried essentially gutting the routine in this file, and the error persisted, so has something to do with the modules being used?

[rljacob]

This might be related to the recent addition of CLM 4.5.1. Have you tried with v0.1?

[jeff-cohere]

This apparently also occurs with v13.10 of the PGI compilers (see the post by @douglasjacobsen at https://gist.github.com/douglasjacobsen/b5dd3ef7b6b720b92a81). Does the PGI compiler have flags that dump more verbose output, suitable for reporting bugs of this sort? We can probably push on them, since a number of DOE machines have purchased licenses.

[worleyph]

Hi Matt,
What is the process for reporting compiler issues? Just send
something to help? This is an abort within the compiler, not a complaint
about bad code. It is also very persistent - even emptying the file in
question except for the module includes and the subroutine declaration
does not eliminate the problem.

Pat

On 12/10/14 11:47 AM, Jeffrey Johnson wrote:

This apparently also occurs with v13.10 of the PGI compilers (see the post by @douglasjacobsen at https://gist.github.com/douglasjacobsen/b5dd3ef7b6b720b92a81). Does the PGI compiler have flags that dump more verbose output, suitable for reporting bugs of this sort? We can probably push on them, since a number of DOE machines have purchased licenses.

---

Reply to this email directly or view it on GitHub:
#62 (comment)

[douglasjacobsen]

I'll try @rljacob's suggestion of v0.1 (I'm actually going to locally revert the commit that brought in CLM 4.5.1, but that should let us know if that was the issue).

[worleyph]

Help is a reasonable place to start. If they need more info, they can ask. I'd just suggest having a reliable reproducer on as few nodes as possible before sending it.

-Matt

---

Matthew R. Normanhttp://users.nccs.gov/~imn/
Computational Climate Scientist
Scientific Computing Group
National Center for Computational Scienceshttps://www.olcf.ornl.gov/
Oak Ridge National Laboratoryhttp://www.ornl.gov/

From: Patrick Worley <worleyph@ornl.govmailto:worleyph@ornl.gov>
Date: Wednesday, December 10, 2014 at 11:53 AM
To: "reply@reply.github.commailto:reply@reply.github.com" <reply@reply.github.commailto:reply@reply.github.com>, "Norman, Matthew R." <normanmr@ornl.govmailto:normanmr@ornl.gov>
Subject: Re: [ACME] Fatal PGI compiler bug when building B case with CLM4.5 on Titan (#62)

Hi Matt,
What is the process for reporting compiler issues? Just send something to help? This is an abort within the compiler, not a complaint about bad code. It is also very persistent - even emptying the file in question except for the module includes and the subroutine declaration does not eliminate the problem.

Pat

On 12/10/14 11:47 AM, Jeffrey Johnson wrote:

This apparently also occurs with v13.10 of the PGI compilers (see the post by @douglasjacobsen at https://gist.github.com/douglasjacobsen/b5dd3ef7b6b720b92a81). Does the PGI compiler have flags that dump more verbose output, suitable for reporting bugs of this sort? We can probably push on them, since a number of DOE machines have purchased licenses.

---

Reply to this email directly or view it on GitHub:
#62 (comment)

[douglasjacobsen]

So, this command at least lets me build now: git revert 114fbe8 -m 1

[jedbrown]

@worleyph When I report compiler bugs, I make a reasonable attempt to reduce and submit what they need to reproduce independently (e.g., a tarball or source+makefile). Commercial compilers have always been slower to act and less transparent in my experience, but a little up-front effort goes a long ways toward getting prompt action.

[jedbrown]

@douglasjacobsen That's a 117 kLOC merge that is needed for further land group development. We need to narrow it down more.

[douglasjacobsen]

@jedbrown yeah, this was just a starting place. I also don't know if I can build with the other two compilers with that merge reverted.

[rljacob]

As Jed noted, that was a huge merge. If you were not yet done porting ACME to your platform or suddenly having compiler errors, its probably best to go back to one commit before 114fbe8.

[douglasjacobsen]

I was not done porting ACME to my platform, but the bug report is actually from titan where the machine files were already finalized.

But even if I went back to 114fbe8 once the machine files are merged into master the ICLM45BGC compset won't build with pgi.

So far, it seems like it's actually the code b7663ce that causes the issue, but I'm testing to make sure it's that one. But that commit is the biggest one from the merge. I'll see if I can get anywhere at determining what causes the issue, but no guarantees.

[jedbrown]

@rljacob That is a stop-gap measure and I think we should try to avoid wasting time working with that old version (at least anyone related to the land model). Is it an option for people encountering this problem to use a different compiler in the short term? Otherwise we should find a work-around for PGI. Did CSEG run into the same problem?

@douglasjacobsen That commit is pretty direct from CESM and if the bug is present there, it indicates that it wasn't introduced in a bad merge. It also means that was likely also a problem for CESM, so maybe they have a work-around or have already reported the bug to PGI.

[douglasjacobsen]

@jedbrown yeah, maybe we should look through their bugzilla and see if we can find something related to PGI version and that version of the code.

[worleyph]

I found the following:

http://www.pgroup.com/userforum/viewtopic.php?t=3691&view=previous&sid=e9dc6dcf687914267f020f67f6ea5b2c

This is from 2008, but it indicates memory as a possible issue. It also
mentions that 'fast' was being specified (even though the user
explicitly set -O0). I have a similar problem - I set -O0 -Mnovect in my
most recent experiments, and I continue to get

pgf90-Info-Switch -Mvect forces -O2

Nowhere do I see -Mvect being used during the compilation, and the
compile log script is pretty verbose. ???

Pat

[worleyph]

From Forrest Hoffman:

"The CMT folks have been fighting with PGI bugs for a couple of months
now, and have spent time developing workarounds. From the traffic I have
seen, I believe bug reports have been filed with PGI, and everyone is
waiting for a new release.

The workarounds CSEG has implemented are not actual CESM bug fixes, and
I don't know what the ACME-CESM collaboration policy is for sharing
temporary workarounds for bugs in compilers. "

[rljacob]

CSEG consolidates their compiler issues on this page: https://wiki.ucar.edu/display/ccsm/Fortran+Compiler+Bug+List

[worleyph]

From Peter Thornton:
"The arrangement that ACME-Land group has with CESM, through our architecture review concalls, is to track and share bug information and fixes between the projects."
So I guess that we need to be talking to the land group more?

[mt5555]

All of CESM and ACME should share bug information and bug fixes between projects. But please dont ask CSEG to fix bugs in ACME.

[worleyph]

From CESG's compiler issues page:

PGI :

Internal compiler error when compiling CLM.

• Exact cause is uncertain, but this may represent a problem with the .mod files produced for certain modules, e.g. TemperatureType.

Occurs with >= 14.1

(Doug saw this with 13.10?)

Fixed in 14.10

[douglasjacobsen]

So, does this imply if a machine doesn't have 14.10 or newer PGI versions, we can't support PGI on those machines?

[rljacob]

I'd say yes. We can't afford to find workarounds for known-to-be-buggy compilers when newer, less-buggy versions of those compilers are available.

[jeff-cohere]

Depending on whether the land group is using the newer "object-oriented" Fortran 200{3,8} features, we might be stuck with very stingy Fortran compiler requirements. Most commercial compilers don't work for this stuff at all except in very recent versions.

[douglasjacobsen]

OK. Do you think we should remove PGI as an optional compiler, and add it back in when there's a working version of the compiler? or leave it in at just tell people that it doesn't work (via the configuration management page?)

[worleyph]

If we are not building clm4.5 this is not an issue. Very few tests
currently use CLM4.5, correct?

[rljacob]

I think machine POC's should indicate that its not supported in the comment column on the row for the PGI compiler for their machine.

ACME is supposed to move to CLM4.5 so we should be trying to build it and/or upgrading our compilers.

[thorntonpe]

Yes, the land code we are starting with is using “object oriented” Fortran 2003 features. We will be using those features for the foreseeable future.

---

Peter Thornton
Climate Change Science Institute / Environmental Sciences Division
Oak Ridge National Laboratory
865-241-3742 (w) or 865-323-2447 (cell)
[CCSI_logo_small]

From: Jeffrey Johnson [mailto:notifications@github.com]
Sent: Wednesday, December 10, 2014 4:29 PM
To: ACME-Climate/ACME
Subject: Re: [ACME] Fatal PGI compiler bug when building B case with CLM4.5 on Titan (#62)

Depending on whether the land group is using the newer "object-oriented" Fortran 200{3,8} features, we might be stuck with very stingy Fortran compiler requirements. Most commercial compilers don't work for this stuff at all except in very recent versions.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-66527214.

[jeff-cohere]

Well, then. Supporting older compilers is out of the question. :-)

[douglasjacobsen]

So, technically the fix for this bug is using a newer PGI version then (i.e. 14.10?), or not using PGI, but it's really a bug that we're not going to fix in any useful way (since we can't)

[thorntonpe]

I’m probably missing something, but it seems like supporting older compilers isn’t a problem, as long as we also support ones that use the fancy new stuff, from 2003… ;)

Peter
From: Jeffrey Johnson [mailto:notifications@github.com]
Sent: Wednesday, December 10, 2014 4:41 PM
To: ACME-Climate/ACME
Cc: Thornton, Peter E.
Subject: Re: [ACME] Fatal PGI compiler bug when building B case with CLM4.5 on Titan (#62)

Well, then. Supporting older compilers is out of the question. :-)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-66529166.

[jeff-cohere]

If a given compiler can't build our source code because our code uses language features that the compiler can't handle, we can't "support" it in any meaningful way.

[worleyph]

Based on my reading of the CSEG bug wiki, this particular issue has
nothing (obvious) to do with a language feature. It was categorized as
being among the f95 bugs.

[douglasjacobsen]

@worleyph sure, but the cause isn't really specified, it even says they aren't sure what the cause is. So it's possible that the cause is due to language features that the compiler fails in a very non elegant way when trying to handle them.

[worleyph]

Everything is possible. Just seemed that the
conversation was taking a path that was not necessarily tied to this
thread. Perhaps a more general discussion on language feature support
should go on Confluence?

[douglasjacobsen]

Sure. @worleyph do you have access to a newer PGI version that you can test with, to see if that fixes the issue? I don't have access to 14.10 (which is what they say fixes the issue) but if we can verify that 14.10 fixes it, then maybe we can close this issue.

[worleyph]

I do not have access to pgi/14.10.0 . I have asked our OLCF POC what the
schedule is for access to newer versions of PGI on Titan (and have not
heard back yet). I also did a quick search and found a November 2014
document indicating that Cray is now providing pgi/14.10.0 on their XC
systems. Unfortunately Edison does not have any PGI support, so it is
unlikely to show up there.

[douglasjacobsen]

@singhbalwinder What PGI version are you using on olympus and sooty? If you have access to 14.10 do you think you can try building / running the ICML45BGC compset on that machine with pgi 14.10?

[worleyph]

May be premature to mention, but I tried both the Cray and Intel compilers for the case
-compset B1850C5L45BGC -res ne30_g16
The Cray build failed (in the land, in clm/src/dyn_subgrid/dynTimeInfoMod.F90). The Intel build completed, but the run died with a seg. fault during the land initialization. I'll try the I case when I get the chance.

[douglasjacobsen]

@worleyph What intel compiler are you using? I have built / run ICLM45BGC with intel 13.1.3 and gnu 4.8.2 so far.

[worleyph]

@douglasjacobsen Used intel/14.0.2.144 . I also have available 12.1.3.293 and 13.1.3.192 on Titan.. I'll try the ICLM45BGC case next, and then the B case with the 13.1.3 version of the Intel compiler. It would be useful to have a B case to test with, but I just "guessed" at one. I have no information that this (B1850C5L45BGC) is supposed to work with ne30_g16 - it simply was in the list of supported compsets. I haven't asked anyone yet either.

[douglasjacobsen]

@worleyph I've built and run the B1850C5CN compset with intel 13.1.3, but that one doesn't use CLM 4.5 as far as I know.

[worleyph]

I tried a prerelease version of PGI version 14.10. 0 on Titan and the build completed successfully. For a B case I then saw a seg. fault during land initialization (same as for the Intel build mentioned above). I was able to get an I case to run successfully. So 14.10.0 does solve the build problem. Next question is whether anyone is using CLM4.5 in a B configuration with CAM5/CAM-SE (with any compiler on any system).