Error in ALM running BGC resolution in debug mode

[jonbob]

In setting up initial tests of the BGC resolution, I created a case using the following command:

create_newcase -case ACME-BGC-stang -compset A_B1850CN -res ne30_oEC -mach mustang -compiler gnu

When I turned the DEBUG mode on, the the env_build.xml file, the run dies during the first land run call with the following error in the cesm log:

ERROR: ERROR in /turquoise/users/jonbob/ACME/components/clm/src/main/subgridAveMod.F90 at line 509
#0 0x2B013425A2F7
#1 0x22C3E79 in __shr_sys_mod_MOD_shr_sys_backtrace
#2 0x22C4270 in __shr_sys_mod_MOD_shr_sys_abort
#3 0x221EAC2 in __shr_assert_mod_MOD_shr_assert
#4 0x221EA79 in __shr_assert_mod_MOD_shr_assert_all_1d
#5 0x161517D in __subgridavemod_MOD_p2g_1d
#6 0x15A2509 in __histfilemod_MOD_hist_update_hbuf_field_1d at histFileMod.F90:0
#7 0x15A3DB0 in __histfilemod_MOD_hist_update_hbuf
#8 0x1546AFE in __clm_driver_MOD_clm_drv
#9 0x153361C in __lnd_comp_mct_MOD_lnd_run_mct
#10 0x42C588 in __component_mod_MOD_component_run
#11 0x40F509 in cesm_comp_mod_MOD_cesm_run
#12 0x4279FF in MAIN at cesm_driver.F90:0

[rljacob]

which compiler and compiler version?

[jonbob]

good point...

Using gnu 4.8.2 on LANL IC machine mustang

[rljacob]

Ok. A land group person can try to reproduce this on Blues with the Gnu 5.2 compiler and take it from there. @thorntonpe , feel free to re-assign.

[bishtgautam]

@jonbob : I got the following error when I tried to create the case on Blues.

./create_newcase -case A_B1850CN.ne30_oEC.blues -compset A_B1850CN -res ne30_oEC -mach blues -compiler gnu
-------------------------------------------------------------------------------
For a list of potential issues in the current tag, please point your web browser to:
https://svn-ccsm-models.cgd.ucar.edu/cesm1/known_problems/
-------------------------------------------------------------------------------
set_compset: no match for grid ne30_oEC 
             to see supported grids issue - 
  create_newcase -list grids 
set_compset: exiting

[bishtgautam]

I just realized that I should have been using rljacob/compsets/add-acmebgc branch instead of master

[bishtgautam]

@rljacob : Run on Blues failed with the following error message error while loading shared libraries: libnetcdff.so.6:

I had created the case using the following commands:

git checkout rljacob/compsets/add-acmebgc
cd cime/scripts
./create_newcase -case A_B1850CN.ne30_oEC.blues \
-compset A_B1850CN -res ne30_oEC -mach blues -compiler gnu \
-project ACME
cd A_B1850CN.ne30_oEC.blues
./cesm_setup
./xmlchange HIST_OPTION=nsteps,HIST_N=1
./xmlchange DEBUG=TRUE
./*.build
./*.submit

[jayeshkrishna]

I am trying a build on blues to see if I get the same error.

[jayeshkrishna]

I am getting the same error as @bishtgautam (for the case above on blues + rljacob/compsets/add-acmebgc),

lcrc/project/ACME/jayesh/acme_scratch/A_B1850CN.ne30_oEC.blues/bld/cesm.exe: error while loading shared libraries: libnetcdff.so.6: cannot open shared object file: No such file or directory
...

[jayeshkrishna]

I am working on a possible fix (blues runtime error with netcdf shared libs) and will ping you ( @bishtgautam ) soon.

[bishtgautam]

thanks.

[jayeshkrishna]

Can you see if the following patch works for you? If so I will create a new bug report and fix it.

diff --git a/cime/machines-acme/config_compilers.xml b/cime/machines-acme/config_compilers.xml
index aa8a80d..1d3b07a 100644
--- a/cime/machines-acme/config_compilers.xml
+++ b/cime/machines-acme/config_compilers.xml
@@ -873,6 +873,7 @@ for mct, etc.
   <MPI_PATH MPILIB="mvapich">/soft/mvapich2/2.2b_psm/gnu-5.2/</MPI_PATH>
   <MPI_LIB_NAME MPILIB="mvapich">mpi</MPI_LIB_NAME>
   <ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nc-config --flibs) -llapack -lblas</ADD_SLIBS>
+  <ADD_SLIBS> -Wl,-rpath,$(NETCDFROOT)/lib </ADD_SLIBS>
   <PIO_FILESYSTEM_HINTS>gpfs </PIO_FILESYSTEM_HINTS>
 </compiler>

[bishtgautam]

A new job is now in the queue and will get back to you, once it finishes.

[bishtgautam]

A different error this time:
/software/gcc-4.7.2/lib64/libgfortran.so.3: version GFORTRAN_1.5' not found`

[jayeshkrishna]

I will first try out some cases on master and see the issues persist and then create a new bug report (to continue the discussion)

[jayeshkrishna]

Is this is a gnu-specific issue? blues+pgi should work otherwise.

[bishtgautam]

The CAM code (components/cam/src/physics/cam/micro_mg_cam.F90) failed to compile with PGI on blues. Here is the error I got:

PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   45987 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   45987 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   45987 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46006 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   46006 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   46006 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46012 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   46012 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   46012 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46031 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   46031 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   46031 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46037 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   46037 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   46037 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46056 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   46056 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   46056 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46062 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   46062 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   46062 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46081 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: TEMPLATE does not exist   46081 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_secd: descriptor does not exist   46081 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)
PGF90-S-0000-Internal compiler error. emit_alnd: descriptor does not exist   46087 (/blues/gpfs/home/gbisht/Projects/ACME/SE-Tasks/issue-603/ACME/components/cam/src/physics/cam/micro_mg_cam.F90: 2981)

[jayeshkrishna]

Sorry about that, this (CAM+blues+pgi failure) is an existing issue - #596 .

[bishtgautam]

The compset A_B1850CN does not exist in the master any longer. So, won't fix it.