The FC5L45BGC compset was failed in cori-knl with "column nbalance error"

[minxu74]

ACME tag: v1.0.0-beta.1-130-g7d39622

Case created by: ./create_newcase --case /global/homes/m/minxu/scratch/F_acmev1_enso_camse_clm45bgc_ne30 -res ne30_ne30 -compset FC5L45BGC -mach cori-knl --compiler intel

PES layout: 2560 x 2 , 64 MPI tasks per node.

Error message:
0014: ERROR in CNBalanceCheckMod.F90 at line 396
1863: column nbalance error = -3.511888906373967E-006 178098 3
0832: column nbalance error = -4.293514919911559E-008 78349 3
1023: column nbalance error = -2.734882737515850E-007 96536 3
2369: column nbalance error = -1.600117219243727E-006 222936 3
2501: column nbalance error = -7.471929814815514E-006 233928 3
0943: column nbalance error = -2.751315590089568E-007 88737 3
0943: Latdeg,Londeg = 59.9466134666123 93.7563801427842
0943: begnb = 2.80072383505868
0943: endnb = 2.80073718112814
0943: delta store = 1.334606945935235E-005
0943: input mass = 1.307093991055480E-005
0943: output mass = 2.010211410065240E-012
0943: net flux = 1.307093790034339E-005
0943: denit = 0.000000000000000E+000
0943: n2onit = 2.010114779247898E-012
0943: no3 leach = 2.137917904780868E-018
0943: no3 runof = 0.000000000000000E+000
0943: ndep = 1.307093991055480E-005
0943: nfix = 0.000000000000000E+000
0943: nsup = 0.000000000000000E+000
0943: fire = 0.000000000000000E+000
0943: dwt = 0.000000000000000E+000
0943: prod = 0.000000000000000E+000

[bishtgautam]

I believe the CLM45%BGC has broken for a very long time. Do you require land BGC to be on for your study?

[minxu74]

@bishtgautam Could you recommend us a workable F-compset with the land biogeochemical cycle? Thanks!

[bishtgautam]

The land team is presently only testing CLM45%CN, but it appears there is no corresponding F compset. Following the definition of FC5L45BGC, you can try adding a new compset FC5L45CN and see if it works.

  <compset>
    <alias>FC5L45CN</alias>
    <lname>2000_CAM5_CLM45%CN_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV</lname>
  </compset>

Btw, I'm not sure if CLM45%CN will meet the scientific requirments of your study.

[minxu74]

@bishtgautam Thanks a lot. I will evaluate the differences between the BGC and CN to see if it fits my research goal. By the way, since those BGCs have broken for a long time, shall we remove them from the config files? Because I used the manage-case to query F-compsets and get a lot of compsets with CLM45%BGC. They should not work. Is it possible to tag those workable or broken compsets?

[bishtgautam]

Even though the CLM45%BGC compset has been broken for awhile (mid-2016 based on #974), at this I'm not in the favor of removing all compsets related to CLM45%BGC unless the land team wants to abandon its support.

Tagging: @thorntonpe, @wjriley

[rljacob]

I think we still have a lot of compset definitions held over from cesm. Not sure all the science groups have gone through and removed the ones not being supported.

[thorntonpe]

I have no interest in or plans for supporting these capabilities, but if BIll or others find them useful I'm not opposed to leaving them in place for the time being. We have many new ALM developments which are not currently protected by tests in the developer suites, and I would rather we direct our time and resources to making a robust testing package for the active and ongoing development efforts.

[minxu74]

Since the error message (N balance error) in this issue is same as that @dmricciuto mentioned in the open issue #974, I would like to close it to avoid the duplicates. Thanks for all suggestions and comments.

[minxu74]

Same error reported by @worleyph at the issue #1379 but with different land physics CN. The error was gone if I ran the model in a pure MPI process i.e. changing the number of threads from 2 to 1 following his suggestion. Since those issues are related, I reopened mine.