Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix crash issue in MOM_SIS when compiling with openmp #6

Merged
merged 1 commit into from
Mar 18, 2014

Conversation

Zhi-Liang
Copy link
Contributor

Using tikal_201304 and MOM6 dev/master, all MOM6_SIS test cases crash when compiled with -openmp with:

FATAL from PE 30: NaN in input field of reproducing_sum(_2d).

This is the xml and fre commands to reproduce the issue:

/ncrc/home2/Niki.Zadeh/frerts/tikal_201403_devmaster20140313/_0/MOM6_SIS.xml.20140313201957

fremake --execute --nolink -x /ncrc/home2/Niki.Zadeh/frerts/tikal_201403_devmaster20140313/_0/MOM6_SIS.xml.20140313201957 -p ncrc2.intel -t repro-openmp MOM6_SIS_libs_compile

fremake --execute -x /ncrc/home2/Niki.Zadeh/frerts/tikal_201403_devmaster20140313/_0/MOM6_SIS.xml.20140313201957 -p ncrc2.intel -t repro-openmp MOM6_SIS_compile

frerun -s -r basic -u --no-transfer --partition t1 -x /ncrc/home2/Niki.Zadeh/frerts/tikal_201403_devmaster20140313/_0/MOM6_SIS.xml.20140313201957 -p ncrc2.intel -t repro-openmp MOM6_GOLD_SIS

Or you can try the following command to build and run the test:

/ncrc/home2/Niki.Zadeh/fre/fre-commands/bin/frerts_batch.csh -p ncrc2.intel -t repro-openmp --release tikal_201403 --fre_stem tikal_201403_devmaster20140313 --mom_git_tag 'dev\/master' -x /ncrc/home2/Niki.Zadeh/xmls/tikal_f1/xml/MOM6_SIS.xml --frerts_ops '--no_rts,--compile,-l,MOM6_SIS_libs_compile,--fre_ops,-u;--no-transfer;--partition=t1,--do_frecheck,--reference_tag,tikal_mom6_2014.01.17' --fre_version 'fre\/bronx-7' --mom_git_tag 'dev\/master' --debuglevel _0 --interactive
MOM6_GOLD_SIS

These the stdouts for openmp and regular runs:

/lustre/f1/Niki.Zadeh/tikal_201403_devmaster20140313_0/MOM6_GOLD_SIS/ncrc2.intel-repro-openmp/stdout/run/MOM6_GOLD_SIS_1x0m20d_32pe.o5031341
/lustre/f1/Niki.Zadeh/tikal_201403_devmaster20140313_0/MOM6_GOLD_SIS/ncrc2.intel-repro/stdout/run/MOM6_GOLD_SIS_1x0m20d_32pe.o5031270

@nikizadehgfdl
Copy link
Contributor

I have tested this branch. It resolves the crash issues for ocean_ice examples and preserves the timestats when compiled with -openmp.
However the ocean_solo test cases are still crashing:
mom6_solo_DOME, mom6_solo_nonBous_global, mom6_solo_Phillips_2layer, mom6_solo_circle_obcs
The following test case runs but does not reproduce non-openmp timestats:
mom6_solo_single_column_z_BATS

Since this branch fixes the problems with ocean_ice experiments I am going to push it to dev/master

nikizadehgfdl added a commit that referenced this pull request Mar 18, 2014
fix crash issue in MOM_SIS when compiling with openmp
@nikizadehgfdl nikizadehgfdl merged commit c97b199 into dev/master Mar 18, 2014
nikizadehgfdl referenced this pull request in nikizadehgfdl/MOM6 Oct 9, 2017
Draft of frepp files for CM2G63L
They worked for CM4i
nikizadehgfdl referenced this pull request in nikizadehgfdl/MOM6 Oct 9, 2017
- NOAA-GFDL/MOM6@ae22a0f Merge pull request mom-ocean#578 from NCAR/dev/ncar
- NOAA-GFDL/MOM6@f4f882f Merge pull request mom-ocean#23 from gustavo-marques/combine_mct_modules
- NOAA-GFDL/MOM6@a85b728 Merge MCT modules
- NOAA-GFDL/MOM6@ba5b5ba Merge pull request mom-ocean#22 from gustavo-marques/check_fluxes
- NOAA-GFDL/MOM6@f5a8483 Adds option to write restart files
- NOAA-GFDL/MOM6@8745b4b Fills additional IOB fluxes
- NOAA-GFDL/MOM6@f6b22ff Merge branch 'dev/ncar' into check_fluxes
- NOAA-GFDL/MOM6@369d5d6 Merge pull request mom-ocean#21 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@2582e0a Adds run paramaters to control SW decomposition
- NOAA-GFDL/MOM6@58f78b9 Fill IOB, some fluxes still need to be checked
- NOAA-GFDL/MOM6@06a30ab Fixed clock and timing
- NOAA-GFDL/MOM6@1bca79b Merge pull request mom-ocean#20 from gustavo-marques/fix_merge
- NOAA-GFDL/MOM6@fb89b08 Adds code that got lost during merge
- NOAA-GFDL/MOM6@7480492 Merge pull request mom-ocean#19 from gustavo-marques/new_merge
- NOAA-GFDL/MOM6@be1e6cf Merge branch 'dev/master' into dev/ncar
- NOAA-GFDL/MOM6@76565cb Merge pull request mom-ocean#17 from gustavo-marques/fix_ssh_slopes
- NOAA-GFDL/MOM6@c04f081 Fix ssh slope calculation
- NOAA-GFDL/MOM6@17d4586 Merge pull request mom-ocean#16 from adcroft/ncar-cherry-picked-fixes
- NOAA-GFDL/MOM6@be46fee (*)Fixes gprime(1) when no layer coordinates are set
- NOAA-GFDL/MOM6@fccb7ae Avoids a SEGV when OBC are not in use with bi-harmonic friction
- NOAA-GFDL/MOM6@f42c970 Merge pull request mom-ocean#15 from gustavo-marques/small_fixes
- NOAA-GFDL/MOM6@31787c2 Adds reference to set_time that was lost during a conflict handling
- NOAA-GFDL/MOM6@d77936e Fix intent(inout) in ice_ocean_boundary
- NOAA-GFDL/MOM6@b403edd Merge pull request mom-ocean#14 from gustavo-marques/docs
- NOAA-GFDL/MOM6@4d346ca Merge branch 'dev/ncar' into docs
- NOAA-GFDL/MOM6@f7c26a5 Clean the module and start doxygenizing it
- NOAA-GFDL/MOM6@cd58e2a Merge pull request mom-ocean#13 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@37a53bc reorganize file structure
- NOAA-GFDL/MOM6@e6c5ddd Merge pull request mom-ocean#12 from gustavo-marques/fill_ice_ocean_bnd
- NOAA-GFDL/MOM6@6b6289a Remove trailing whitespace
- NOAA-GFDL/MOM6@7ce40e5 Fix undefined reference to fill_ice_ocean_bnd
- NOAA-GFDL/MOM6@d517719 Merge branch 'dev/ncar' into fill_ice_ocean_bnd
- NOAA-GFDL/MOM6@b69ab33 Filling ice_ocean_bnd - Draft
- NOAA-GFDL/MOM6@a306c90 Merge pull request mom-ocean#11 from adcroft/use-ocn-public-for-surface
- NOAA-GFDL/MOM6@c83e238 Correct indexing when using ocean_public_type
- NOAA-GFDL/MOM6@1f39c84 Replaces use of "surface" with "ocean_public_type"
- NOAA-GFDL/MOM6@5cd8d7a Filling ice_ocean_bnd, still needs work
- NOAA-GFDL/MOM6@3ceef26 Merge pull request mom-ocean#10 from adcroft/sketch-out-mct-run
- NOAA-GFDL/MOM6@714e8d6 Adds placeholders for stepping foward MOM
- NOAA-GFDL/MOM6@7f2207c Merge pull request #7 from adcroft/fix-mct-whitespace
- NOAA-GFDL/MOM6@a1db77f Cleans up white space
- NOAA-GFDL/MOM6@5700d67 Merge pull request #6 from adcroft/add-use-only-for-imports
- NOAA-GFDL/MOM6@beac3ed Adds "only" for all use statements
- NOAA-GFDL/MOM6@231d19e Merge pull request #5 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@3a923a2 Got initialization working
- NOAA-GFDL/MOM6@8b38141 Wrapped global data
- NOAA-GFDL/MOM6@970f848 added debugging
- NOAA-GFDL/MOM6@2540ed8 Filled empty subroutines in ocn_comp_mct.F90
- NOAA-GFDL/MOM6@ed3ed30 Added ocn_import_export module
- NOAA-GFDL/MOM6@aba7dc2 Completed ocn_init_mct
- NOAA-GFDL/MOM6@5df3bc3 Added method to point ponters to members of ocean_state_type
- NOAA-GFDL/MOM6@57e6086 Added methods to return domain/grid shape
- NOAA-GFDL/MOM6@37e7356 Added ocn_import_export.F90
- NOAA-GFDL/MOM6@2b794da Merge pull request #4 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@fbd405b Added coupler indices module
- NOAA-GFDL/MOM6@fa86163 Merge pull request #3 from NCAR/dev/gfdl
- NOAA-GFDL/MOM6@e8f2999 Merge pull request #2 from gustavo-marques/temp_salt_z_init_file
- NOAA-GFDL/MOM6@86936ba Adds option to read initial temp and salt from separate files
- NOAA-GFDL/MOM6@54f7976 Most initialization working
- NOAA-GFDL/MOM6@426357c Time and calendar initialization in ocn_comp_mct.F90
- NOAA-GFDL/MOM6@346097d Moved blank ocn_comp_mct.F90
- NOAA-GFDL/MOM6@405fecb Merge pull request #1 from gustavo-marques/ice_shelf
- NOAA-GFDL/MOM6@711a613 Removes frazil's contribution to evaporation
nikizadehgfdl referenced this pull request in nikizadehgfdl/MOM6 Oct 9, 2017
- Updates MOM_parameter_doc files.

- NOAA-GFDL/MOM6@cde6d3f Merge branch 'dev/master' into dev/gfdl
- NOAA-GFDL/MOM6@ae22a0f Merge pull request mom-ocean#578 from NCAR/dev/ncar
- NOAA-GFDL/MOM6@f4f882f Merge pull request mom-ocean#23 from gustavo-marques/combine_mct_modules
- NOAA-GFDL/MOM6@a85b728 Merge MCT modules
- NOAA-GFDL/MOM6@ba5b5ba Merge pull request mom-ocean#22 from gustavo-marques/check_fluxes
- NOAA-GFDL/MOM6@f5a8483 Adds option to write restart files
- NOAA-GFDL/MOM6@8745b4b Fills additional IOB fluxes
- NOAA-GFDL/MOM6@f6b22ff Merge branch 'dev/ncar' into check_fluxes
- NOAA-GFDL/MOM6@369d5d6 Merge pull request mom-ocean#21 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@2582e0a Adds run paramaters to control SW decomposition
- NOAA-GFDL/MOM6@58f78b9 Fill IOB, some fluxes still need to be checked
- NOAA-GFDL/MOM6@06a30ab Fixed clock and timing
- NOAA-GFDL/MOM6@1bca79b Merge pull request mom-ocean#20 from gustavo-marques/fix_merge
- NOAA-GFDL/MOM6@fb89b08 Adds code that got lost during merge
- NOAA-GFDL/MOM6@7480492 Merge pull request mom-ocean#19 from gustavo-marques/new_merge
- NOAA-GFDL/MOM6@be1e6cf Merge branch 'dev/master' into dev/ncar
- NOAA-GFDL/MOM6@76565cb Merge pull request mom-ocean#17 from gustavo-marques/fix_ssh_slopes
- NOAA-GFDL/MOM6@c04f081 Fix ssh slope calculation
- NOAA-GFDL/MOM6@17d4586 Merge pull request mom-ocean#16 from adcroft/ncar-cherry-picked-fixes
- NOAA-GFDL/MOM6@be46fee (*)Fixes gprime(1) when no layer coordinates are set
- NOAA-GFDL/MOM6@fccb7ae Avoids a SEGV when OBC are not in use with bi-harmonic friction
- NOAA-GFDL/MOM6@f42c970 Merge pull request mom-ocean#15 from gustavo-marques/small_fixes
- NOAA-GFDL/MOM6@31787c2 Adds reference to set_time that was lost during a conflict handling
- NOAA-GFDL/MOM6@d77936e Fix intent(inout) in ice_ocean_boundary
- NOAA-GFDL/MOM6@b403edd Merge pull request mom-ocean#14 from gustavo-marques/docs
- NOAA-GFDL/MOM6@4d346ca Merge branch 'dev/ncar' into docs
- NOAA-GFDL/MOM6@f7c26a5 Clean the module and start doxygenizing it
- NOAA-GFDL/MOM6@cd58e2a Merge pull request mom-ocean#13 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@37a53bc reorganize file structure
- NOAA-GFDL/MOM6@e6c5ddd Merge pull request mom-ocean#12 from gustavo-marques/fill_ice_ocean_bnd
- NOAA-GFDL/MOM6@6b6289a Remove trailing whitespace
- NOAA-GFDL/MOM6@7ce40e5 Fix undefined reference to fill_ice_ocean_bnd
- NOAA-GFDL/MOM6@d517719 Merge branch 'dev/ncar' into fill_ice_ocean_bnd
- NOAA-GFDL/MOM6@b69ab33 Filling ice_ocean_bnd - Draft
- NOAA-GFDL/MOM6@a306c90 Merge pull request mom-ocean#11 from adcroft/use-ocn-public-for-surface
- NOAA-GFDL/MOM6@c83e238 Correct indexing when using ocean_public_type
- NOAA-GFDL/MOM6@1f39c84 Replaces use of "surface" with "ocean_public_type"
- NOAA-GFDL/MOM6@5cd8d7a Filling ice_ocean_bnd, still needs work
- NOAA-GFDL/MOM6@3ceef26 Merge pull request mom-ocean#10 from adcroft/sketch-out-mct-run
- NOAA-GFDL/MOM6@714e8d6 Adds placeholders for stepping foward MOM
- NOAA-GFDL/MOM6@7f2207c Merge pull request #7 from adcroft/fix-mct-whitespace
- NOAA-GFDL/MOM6@a1db77f Cleans up white space
- NOAA-GFDL/MOM6@5700d67 Merge pull request #6 from adcroft/add-use-only-for-imports
- NOAA-GFDL/MOM6@beac3ed Adds "only" for all use statements
- NOAA-GFDL/MOM6@231d19e Merge pull request #5 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@3a923a2 Got initialization working
- NOAA-GFDL/MOM6@8b38141 Wrapped global data
- NOAA-GFDL/MOM6@970f848 added debugging
- NOAA-GFDL/MOM6@2540ed8 Filled empty subroutines in ocn_comp_mct.F90
- NOAA-GFDL/MOM6@ed3ed30 Added ocn_import_export module
- NOAA-GFDL/MOM6@aba7dc2 Completed ocn_init_mct
- NOAA-GFDL/MOM6@5df3bc3 Added method to point ponters to members of ocean_state_type
- NOAA-GFDL/MOM6@57e6086 Added methods to return domain/grid shape
- NOAA-GFDL/MOM6@37e7356 Added ocn_import_export.F90
- NOAA-GFDL/MOM6@2b794da Merge pull request #4 from alperaltuntas/dev/ncar
- NOAA-GFDL/MOM6@fbd405b Added coupler indices module
- NOAA-GFDL/MOM6@fa86163 Merge pull request #3 from NCAR/dev/gfdl
- NOAA-GFDL/MOM6@e8f2999 Merge pull request #2 from gustavo-marques/temp_salt_z_init_file
- NOAA-GFDL/MOM6@86936ba Adds option to read initial temp and salt from separate files
- NOAA-GFDL/MOM6@54f7976 Most initialization working
- NOAA-GFDL/MOM6@426357c Time and calendar initialization in ocn_comp_mct.F90
- NOAA-GFDL/MOM6@346097d Moved blank ocn_comp_mct.F90
- NOAA-GFDL/MOM6@405fecb Merge pull request #1 from gustavo-marques/ice_shelf
- NOAA-GFDL/MOM6@711a613 Removes frazil's contribution to evaporation
nikizadehgfdl referenced this pull request in nikizadehgfdl/MOM6 Oct 9, 2017
Add 'mfo' (mass transport through straits' to refine_diag
gustavo-marques referenced this pull request in gustavo-marques/MOM6 Sep 23, 2019
marshallward pushed a commit that referenced this pull request Dec 19, 2020
Change default for USE_NET_FW_ADJUSTMENT_SIGN_BUG in drivers
SMoorthi-emc pushed a commit to SMoorthi-emc/MOM6 that referenced this pull request Mar 2, 2021
hmkhatri pushed a commit to hmkhatri/MOM6 that referenced this pull request Mar 5, 2021
…-2021-02-19

Dev gfdl main candidate 2021 02 19
marshallward referenced this pull request in Hallberg-NOAA/MOM6 Nov 25, 2021
Add initialization tests using CS%initialized
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants