NorESM2-LM/CAM6 crashing unexpectedly (shr_reprosum_calc error) #420
-
|
Hello,I am trying to run a NorESM2-LM atmosphere only simulation with NF2000fsstfrc2 compset with cyclic year as 2018. It uses an earlier SSP245 restart file from NSSP245frc2_03_f09_tn14_20200630. The simulation is kept for running for 20 years, it ran succesfully for the first year 2018, giving monthly output files. It also generated the annual restart file and crashed unexpectedly giving the below error. When I restarted the simulation with the new restart file of 2019, it again crashed after running for 5 months. Are there anyone familiar with these kind of errors? I have seen people reported the same for CESM2, but those solutions are not working for my case. Opened file NF2018fsst_f09_f09_mg17_20221207.cam.h0.2019-05.nc to write |
Beta Was this translation helpful? Give feedback.
Replies: 9 comments 6 replies
-
|
@AnushaNTNU , Could you share yours CASE, run folder and NorESM source code ? Also, did you created this compset self or is it coming from NorESM? |
Beta Was this translation helpful? Give feedback.
-
|
Yes |
Beta Was this translation helpful? Give feedback.
-
|
@AnushaNTNU could you copy yours CASE folder to work directory as it seems like that there is something wrong with configuration as in ice*log file there are NaN from beginnings. Please also copy the Sourcecode to WORK folder. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I copied the case folder and cam source code here : /cluster/work/users/a2021/noresm/NF2018fsst_f09_f09_mg17_20221 |
Beta Was this translation helpful? Give feedback.
-
|
@AnushaNTNU |
Beta Was this translation helpful? Give feedback.
-
|
Hi Dirk
Thanks for checking it
Yes, I still have the problem
I tried resubmitting it multiple times. It runs for some time (an year and 5 months precisely) and stops with the error '
Opened file NF2018fsst_f09_f09_mg17_20221207.cam.h0.2019-05.nc to write
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 9
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 10
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 137
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 266
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 456
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 11
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 139
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 264
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs on process 460
ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
Since we have used all new files for the atmospheric compset of 2018, we are checking them now. But the error persists.
Thanks
Anusha
…________________________________
From: DirkOlivie ***@***.***>
Sent: Tuesday, January 24, 2023 6:24 PM
To: NorESMhub/NorESM ***@***.***>
Cc: Anusha Sathyanadh ***@***.***>; Mention ***@***.***>
Subject: Re: [NorESMhub/NorESM] NorESM2-LM/CAM6 crashing unexpectedly (shr_reprosum_calc error) (Discussion #420)
@AnushaNTNU<https://github.com/AnushaNTNU>
do you still have the same problem?
I have checked a fixed-SST simulation of mine, and the ice*log file also contains NaNs - so I think that is not the reason for your simulation to stop. Did you try to resubmit your simulation recently?
Best regards, Dirk
—
Reply to this email directly, view it on GitHub<#420 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYD63O75PMJYRTKGYYHKBITWUAF3VANCNFSM6AAAAAATW6GLEE>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Anusha, et al.,
this is a dycore error that typically arises when forcing is very sudden
(or strong, or wrong) and the time-step ends up being too long for the
dycore to handle.
I would first look at the input, especially the SST and ice cover data,
to check if there may be file/data corruption and, if not, whether there
are unrealistic data (e.g. a sudden change in sea-ice cover where
previously ice-covered ocean gets suddenly exposed). If the crash occurs
mid-month, this is the most likely explanation.
If that looks OK, you may try testing your run by switching off all the
AM fixer code, i.e. setting the namelist variable fv_am_correction,
fv_am_diag, fv_am_fix_lbl, fv_am_fixer, fv_am_geom_crrct all to .false.
This will not solve your problem but the model will crash somewhere else
which might help you diagnose the problem.
If the input looks OK and the dycore still shows up as the place where
the problem originates, then try to shorten the dycore time-step by
specifying fv_nsplit, fv_nspltrac, or increasing fv_nspltvrm (note that
if you specify fv_nsplit and fv_nspltrac as /=0, they must both be
multiples of fv_nspltvrm, and fv_nsplit must be a multiple of fv_nspltrac).
Best regards
Thomas
Thomas Toniazzo
(NORCE, Bergen, Norway)
…On 2023-01-25 09:15, AnushaNTNU wrote:
Hi Dirk
Thanks for checking it
Yes, I still have the problem
I tried resubmitting it multiple times. It runs for some time (an year
and 5 months precisely) and stops with the error '
Opened file NF2018fsst_f09_f09_mg17_20221207.cam.h0.2019-05.nc to write
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 9
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 10
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 137
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 266
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 456
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 11
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 139
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 264
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 460
ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
Since we have used all new files for the atmospheric compset of 2018, we
are checking them now. But the error persists.
Thanks
Anusha
________________________________
From: DirkOlivie ***@***.***>
Sent: Tuesday, January 24, 2023 6:24 PM
To: NorESMhub/NorESM ***@***.***>
Cc: Anusha Sathyanadh ***@***.***>; Mention ***@***.***>
Subject: Re: [NorESMhub/NorESM] NorESM2-LM/CAM6 crashing unexpectedly
(shr_reprosum_calc error) (Discussion #420)
@AnushaNTNU<https://github.com/AnushaNTNU>
do you still have the same problem?
I have checked a fixed-SST simulation of mine, and the ice*log file also
contains NaNs - so I think that is not the reason for your simulation to
stop. Did you try to resubmit your simulation recently?
Best regards, Dirk
—
Reply to this email directly, view it on
GitHub<#420 (comment)>,
or
unsubscribe<https://github.com/notifications/unsubscribe-auth/AYD63O75PMJYRTKGYYHKBITWUAF3VANCNFSM6AAAAAATW6GLEE>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
—
Reply to this email directly, view it on GitHub
<#420 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADZGLJGRL2EBPVCFTNNZR7DWUDOJHANCNFSM6AAAAAATW6GLEE>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Thomas
Thanks a lot. Thats very clear.
I didn't find any issues with the SST ,seaice data so far as we are just using one year of data (2018) as input
Let me look at the other input data one by one and follow your suggestions to diagnose the exact issue
Thanks,
Anusha
________________________________
From: tto061 ***@***.***>
Sent: Wednesday, January 25, 2023 11:21 AM
To: NorESMhub/NorESM ***@***.***>
Cc: Anusha Sathyanadh ***@***.***>; Mention ***@***.***>
Subject: Re: [NorESMhub/NorESM] NorESM2-LM/CAM6 crashing unexpectedly (shr_reprosum_calc error) (Discussion #420)
Hi Anusha, et al.,
this is a dycore error that typically arises when forcing is very sudden
(or strong, or wrong) and the time-step ends up being too long for the
dycore to handle.
I would first look at the input, especially the SST and ice cover data,
to check if there may be file/data corruption and, if not, whether there
are unrealistic data (e.g. a sudden change in sea-ice cover where
previously ice-covered ocean gets suddenly exposed). If the crash occurs
mid-month, this is the most likely explanation.
If that looks OK, you may try testing your run by switching off all the
AM fixer code, i.e. setting the namelist variable fv_am_correction,
fv_am_diag, fv_am_fix_lbl, fv_am_fixer, fv_am_geom_crrct all to .false.
This will not solve your problem but the model will crash somewhere else
which might help you diagnose the problem.
If the input looks OK and the dycore still shows up as the place where
the problem originates, then try to shorten the dycore time-step by
specifying fv_nsplit, fv_nspltrac, or increasing fv_nspltvrm (note that
if you specify fv_nsplit and fv_nspltrac as /=0, they must both be
multiples of fv_nspltvrm, and fv_nsplit must be a multiple of fv_nspltrac).
Best regards
Thomas
Thomas Toniazzo
(NORCE, Bergen, Norway)
On 2023-01-25 09:15, AnushaNTNU wrote:
Hi Dirk
Thanks for checking it
Yes, I still have the problem
I tried resubmitting it multiple times. It runs for some time (an year
and 5 months precisely) and stops with the error '
Opened file NF2018fsst_f09_f09_mg17_20221207.cam.h0.2019-05.nc to write
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 9
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 10
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 137
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 266
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 456
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 11
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 139
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 264
SHR_REPROSUM_CALC: Input contains 0.20000E+01 NaNs and 0.00000E+00 INFs
on process 460
ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
Since we have used all new files for the atmospheric compset of 2018, we
are checking them now. But the error persists.
Thanks
Anusha
________________________________
From: DirkOlivie ***@***.***>
Sent: Tuesday, January 24, 2023 6:24 PM
To: NorESMhub/NorESM ***@***.***>
Cc: Anusha Sathyanadh ***@***.***>; Mention ***@***.***>
Subject: Re: [NorESMhub/NorESM] NorESM2-LM/CAM6 crashing unexpectedly
(shr_reprosum_calc error) (Discussion #420)
@AnushaNTNU<https://github.com/AnushaNTNU>
do you still have the same problem?
I have checked a fixed-SST simulation of mine, and the ice*log file also
contains NaNs - so I think that is not the reason for your simulation to
stop. Did you try to resubmit your simulation recently?
Best regards, Dirk
—
Reply to this email directly, view it on
GitHub<#420 (comment)>,
or
unsubscribe<https://github.com/notifications/unsubscribe-auth/AYD63O75PMJYRTKGYYHKBITWUAF3VANCNFSM6AAAAAATW6GLEE>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
—
Reply to this email directly, view it on GitHub
<#420 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADZGLJGRL2EBPVCFTNNZR7DWUDOJHANCNFSM6AAAAAATW6GLEE>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
—
Reply to this email directly, view it on GitHub<#420 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYD63O7GXT7QLNOFSA2Q6BDWUD5BJANCNFSM6AAAAAATW6GLEE>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Anusha I have not checked the input files directly but looking at the output it seems to be a bug in the aerosol emission files. The reported emissions output are 10^3 - 10^4 higher than they should have been and with a different geographical pattern. You can also see it in radiation. The amount of model aerosol is so high that almost no solar radiation reaches the surface. (Sort of Venus-Titan hybrid) Given that problem I guess it is to be expected that the model is having a hard time making sensible fluxes otherwise as well. Øyvind |
Beta Was this translation helpful? Give feedback.
-
|
does the problem still persists? Best regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hi |
Beta Was this translation helpful? Give feedback.
-
|
the 0.9x1.25 files are not available yet on fram/betzy. But I can create them (should not take much effort) and put them on Betzy. Best regards, Dirk |
Beta Was this translation helpful? Give feedback.
-
|
I have now also uploaded the 0.9x1.25 emission files to Betzy (in the same directory as mentioned above). Best regards, Dirk |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot |
Beta Was this translation helpful? Give feedback.
-
|
The problem is solved. Thanks a lot for the help. |
Beta Was this translation helpful? Give feedback.
Hi Anusha
I have not checked the input files directly but looking at the output it seems to be a bug in the aerosol emission files. The reported emissions output are 10^3 - 10^4 higher than they should have been and with a different geographical pattern.
You can also see it in radiation. The amount of model aerosol is so high that almost no solar radiation reaches the surface. (Sort of Venus-Titan hybrid)
Given that problem I guess it is to be expected that the model is having a hard time making sensible fluxes otherwise as well.
Øyvind