Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Floating invalid with several I-cases with DEBUG on cori-knl, such asSMS_D.f19_g16.I1850CNECACNTBC.cori-knl_intel.elm-eca #3123

Closed
ndkeen opened this issue Aug 14, 2019 · 11 comments
Closed

Floating invalid with several I-cases with DEBUG on cori-knl, such asSMS_D.f19_g16.I1850CNECACNTBC.cori-knl_intel.elm-eca #3123

ndkeen opened this issue Aug 14, 2019 · 11 comments
Assignees
@jinyuntang
Labels
Cori Land

Comments

@ndkeen
Copy link
Contributor

ndkeen commented Aug 14, 2019

Using master of Aug12th 2019. There are several I-cases that will fail with what may or may not be same error. The tests run without DEBUG.

Building with intel19, it might be hitting similar error as #3088, which is likely different than above. With intel (version 18), it's writing a lot of longjmp messages, but nothing useful.

/global/cscratch1/sd/ndk/acme_scratch/cori-knl/m07-aug12/SMS_D.f19_g16.I1850CNECACNTBC.cori-knl_intel.clm-eca.r00
@ndkeen
Copy link
Contributor Author

ndkeen commented Dec 5, 2019 •
edited
Loading

This test is still failing with master of Nov 6th.

  8: forrtl: error (65): floating invalid
  8: Image              PC                Routine            Line        Source             
  8: e3sm.exe           00000000067D875E  Unknown               Unknown  Unknown
  8: e3sm.exe           0000000006079D40  Unknown               Unknown  Unknown
  8: e3sm.exe           0000000002577DF3  allocationmod_mp_        2415  AllocationMod.F90
  8: e3sm.exe           0000000003127778  soillittdecompmod         410  SoilLittDecompMod.F90
  8: e3sm.exe           0000000002BD62F9  ecosystemdynmod_m         594  EcosystemDynMod.F90
  8: e3sm.exe           00000000008CF653  clm_driver_mp_clm        1029  clm_driver.F90


         else ! ECA mode or MIC outcompete plant mode                                                                                                     
            do j = 1, nlevdecomp
               do fc=1,num_soilc
                  c = filter_soilc(fc)

                  ! ECA and MIC mode assume mineral surface adsorption flux is a potential competitor of solution P                                       
                  ! assume solutionP - labileP not equilibrate within 30 min, due to instantaneous                                                        
                  ! plant P uptake, microbial P uptake/release                                                                                            
                  ! secondary P desorption is assumed to go into solution P pool                                                                          

                  ! potential adsorption rate without plant and microbial interaction                                                                     
                  ! including weathering, deposition, phosphatase, mineralization, immobilization, plant uptake                                           
                  dsolutionp_dt(c,j) = gross_pmin_vr(c,j) -potential_immob_p_vr(c,j) - &
                       col_plant_pdemand_vr(c,j) + biochem_pmin_vr_col(c,j) + &    ! <-- line 2415 of AllocationMod.F90
                       primp_to_labilep_vr_col(c,j) + pdep_to_sminp(c) *ndep_prof(c,j)
                  adsorb_to_labilep_vr(c,j) = (vmax_minsurf_p_vr(isoilorder(c),j)* km_minsurf_p_vr(isoilorder(c),j)) / &
                       ((km_minsurf_p_vr(isoilorder(c),j)+max(solutionp_vr(c,j),0._r8))**2._r8 ) * dsolutionp_dt(c,j)

@jinyuntang
Copy link
Contributor

jinyuntang commented Dec 5, 2019 via email

@ndkeen
Copy link
Contributor Author

ndkeen commented Dec 5, 2019

It can likely wait until that PR is merged, but would be great if you could add "Fixes #" to the PR if indeed it allows this test to complete.

@jinyuntang
Copy link
Contributor

jinyuntang commented Dec 5, 2019 via email

@ndkeen
Copy link
Contributor Author

ndkeen commented Jul 8, 2020

Using master of Jul 7th, I tried SMS_D.f19_g16.I1850CNECACNTBC.cori-knl_intel.clm-eca and it fails in the same way as above.

@ndkeen
Copy link
Contributor Author

ndkeen commented Nov 24, 2020

Same error with master of Nov19th. Changing to .elm-eca testmod.

@ndkeen ndkeen changed the title Floating invalid with several I-cases with DEBUG on cori-knl, such asSMS_D.f19_g16.I1850CNECACNTBC.cori-knl_intel.clm-eca Floating invalid with several I-cases with DEBUG on cori-knl, such asSMS_D.f19_g16.I1850CNECACNTBC.cori-knl_intel.elm-eca Nov 24, 2020
rljacob pushed a commit that referenced this issue Apr 21, 2021
@jgfouca
Merge pull request #3123 from ESMCI/jgfouca/branch-for-acme-split-201…
540d950 
…9-05-28

Jgfouca/branch for acme split 2019 05 28

Change List:

Add first stage of ocean/ice shelf coupling
Bring COMPOSE semi-Lagrangian tracer transport code into HOMME and E3SM (required minor changes to build.py)
Add checkpoint of timing data immediately after initialization
bless_test_results: Be able to handle build-only tests
Test suite: scripts_regression_tests
Test baseline:
Test namelist changes:
Test status: BFB

Fixes [CIME Github issue #]

User interface changes?:

Update gh-pages html (Y/N)?:

Code review: @jedwards4b
@ndkeen
Copy link
Contributor Author

ndkeen commented Apr 23, 2021

Test still failing, but different line number now: ?

192: forrtl: error (65): floating invalid
192: Image              PC                Routine            Line        Source
192: e3sm.exe           0000000006A68B54  Unknown               Unknown  Unknown
192: e3sm.exe           00000000063B00D0  Unknown               Unknown  Unknown
192: e3sm.exe           000000000255F0AB  allocationmod_mp_        1760  AllocationMod.F90
192: e3sm.exe           0000000003103237  soillittdecompmod         413  SoilLittDecompMod.F90
192: e3sm.exe           0000000002BC47F1  ecosystemdynmod_m         611  EcosystemDynMod.F90
192: e3sm.exe           000000000092459E  elm_driver_mp_elm        1083  elm_driver.F90
192: e3sm.exe           000000000551E6E3  Unknown               Unknown  Unknown
192: e3sm.exe           00000000054C898A  Unknown               Unknown  Unknown
192: e3sm.exe           00000000054C79E1  Unknown               Unknown  Unknown

@ndkeen
Copy link
Contributor Author

ndkeen commented Apr 23, 2021

Might be same issue as #3786

rljacob pushed a commit that referenced this issue May 6, 2021
@jgfouca
Merge pull request #3123 from ESMCI/jgfouca/branch-for-acme-split-201…
ab1c654 
…9-05-28

Jgfouca/branch for acme split 2019 05 28

Change List:

Add first stage of ocean/ice shelf coupling
Bring COMPOSE semi-Lagrangian tracer transport code into HOMME and E3SM (required minor changes to build.py)
Add checkpoint of timing data immediately after initialization
bless_test_results: Be able to handle build-only tests
Test suite: scripts_regression_tests
Test baseline:
Test namelist changes:
Test status: BFB

Fixes [CIME Github issue #]

User interface changes?:

Update gh-pages html (Y/N)?:

Code review: @jedwards4b
@ndkeen
Copy link
Contributor Author

ndkeen commented Nov 10, 2021 •
edited
Loading

With a Nov 2021 master:

SMS_D.f19_g16.I1850CNECACNTBC.cori-knl_intel yields:

 17: forrtl: error (65): floating invalid
 17: Image              PC                Routine            Line        Source
 17: e3sm.exe           0000000006CC1C54  Unknown               Unknown  Unknown
 17: e3sm.exe           00000000066091D0  Unknown               Unknown  Unknown
 17: e3sm.exe           0000000000E5C64E  subgridavemod_mp_        1075  subgridAveMod.F90
 17: e3sm.exe           00000000029F7701  cnpbudgetmod_mp_c        1135  CNPBudgetMod.F90
 17: e3sm.exe           00000000029F6CD9  cnpbudgetmod_mp_c        1072  CNPBudgetMod.F90
 17: e3sm.exe           000000000091A3E9  elm_driver_mp_elm         578  elm_driver.F90
 17: e3sm.exe           00000000057777E3  Unknown               Unknown  Unknown
 17: e3sm.exe           0000000005721A8A  Unknown               Unknown  Unknown
 17: e3sm.exe           0000000005720AE1  Unknown               Unknown  Unknown
 17: e3sm.exe           0000000005777BCA  Unknown               Unknown  Unknown
 17: e3sm.exe           0000000006604129  Unknown               Unknown  Unknown
 17: e3sm.exe           0000000006EE2D7F  Unknown               Unknown  Unknown

Same error with SMS_D_PMx1.f19_g16.I1850CNECACNTBC.cori-knl_intel

And then trying with GNU, the ERS debug test fails differently (likely will be same fail for SMS):
ERS_D.f19_g16.I1850CNECACNTBC.cori-knl_gnu

  0: At line 1051 of file /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/components/elm/src/biogeochem/CNPBudgetMod.F90
  0: Fortran runtime error: Index '20' of dimension 1 of array 'budg_statel' above upper bound of 19
  0:
  0: Error termination. Backtrace:
  0: #0  0x12ccaaa in restart_read
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/components/elm/src/biogeochem/CNPBudgetMod.F90:1051
  0: #1  0x12dd030 in __cnpbudgetmod_MOD_cnpbudget_restart
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/components/elm/src/biogeochem/CNPBudgetMod.F90:487
  0: #2  0x842c73 in __restfilemod_MOD_restfile_read
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/components/elm/src/main/restFileMod.F90:695
  0: #3  0x64f735 in __elm_initializemod_MOD_initialize2
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/components/elm/src/main/elm_initializeMod.F90:757
  0: #4  0x612be3 in __lnd_comp_mct_MOD_lnd_init_mct
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/components/elm/src/cpl/lnd_comp_mct.F90:278
  0: #5  0x4498a3 in __component_mod_MOD_component_init_cc
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/driver-mct/main/component_mod.F90:248
  0: #6  0x42aaeb in __cime_comp_mod_MOD_cime_init
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/driver-mct/main/cime_comp_mod.F90:1431
  0: #7  0x441969 in cime_driver
  0:    at /global/cscratch1/sd/ndk/wacmy/ndk_machinefiles_add-compiler-flag-for-one-mpas-source-with-gnu/driver-mct/main/cime_driver.F90:122

@ndkeen ndkeen mentioned this issue Nov 10, 2021
Closed
@ndkeen ndkeen mentioned this issue Mar 2, 2022
Closed
@ndkeen
Copy link
Contributor Author

ndkeen commented Aug 16, 2022

See some of the same issues with different land case: #5138

@ndkeen
Copy link
Contributor Author

ndkeen commented Aug 16, 2022

Will close in favor of #5138

@ndkeen ndkeen closed this as completed Aug 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jinyuntang jinyuntang

Labels
Cori Land
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rljacob @ndkeen @jinyuntang