Issue with land cells in MOM6-SIS2 coupled mode

[zi-hua-liu]

I am currently configuring regional MOM6 with four open boundaries, implying an absence of land for my model setup. The ESMG-gridtools was employed to generate the horizontal grids. However, the model crashes during the initialization, as indicated by the following error message: "FATAL from PE 0: NETCDF ERROR: NetCDF: Variable not found File=INPUT/atmos_mosaic_tile1Xland_mosaic_tile1.ncField=ncells"

I checked the file "atmos_mosaic_tile1Xland_mosaic_tile1.nc" and it displays zero 'ncells'. This has also been checked through the regeneration of the mosaic files using FRE-NCtools. But the issue persists, and I continue to receive the same error message.

I have a couple of queries regarding this:

• In a model configuration lacking land, should the 'ncells' in the coupler grids between the atmosphere and land components indeed be zero? The grid files for landXocean and atmosXocean seem correct, with 'ncells' corresponding to the number of ocean cells.

• Is it mandatory for the MOM6-SIS2 coupled model to include some land cells?"

[yichengt900]

@zi-hua-liu , Thank you for raising this issue and providing detailed information. Below are my responses to your questions:

1. Your interpretation is correct. If there is no land portion in your model grid, the ncells in atmos_mosaic_tile1Xland_mosaic_tile1.nc should be zero.

2. The short answer is no. I did not encounter the error message you mentioned in your post when I tried a small domain with all ocean grid cells and four OBCS.

To investigate the issue further, could you please share your model configurations (e.g., grid files, obc files, model setup files) so I can attempt to replicate the error on my end? Additionally, could you provide details on how you compile your model and which source codes (GitHub repo, tag, etc.) you are using? This information will help me determine the root cause. Thanks.

[zi-hua-liu]

@yichengt900 , Thank you very much for the answer and willing to help me further checking the details! I appreciate it!

I have uploaded all configuration files for my model to Google Drive due to their large size (160M). After unzipping, you will find a folder named 'INPUT'. Inside, there is a bash script 'momrun.sh' which I utilized for job submission. Another directory, also named 'INPUT', contains all necessary NetCDF files, including grids, boundary conditions, initial conditions, and forcing data.

For the compilation, I used 'poseidon-intel.mk' as the template with the Intel compiler on the WHOI HPC system.

I compiled the Ocean-SIS2 model according to the instructions provided in the GFDL MOM6-examples 'Getting-started' page. This involved "compiling the FMS shared code" followed by "compiling MOM6 in the MOM6-SIS2 coupled mode". With this, I can get the Ocean-SIS2 coupled mode configuration for MOM6. Regarding the version of MOM6, I am not sure how to find out the exact version but I think it might be the latest, as I cloned the core repositories also from the 'Getting-started' page less than two months ago.

I hope this includes all the necessary information for a test. I am more than willing to provide any additional details if needed. Thank you!

[yichengt900]

@zi-hua-liu, thank you once more for providing all the details. This makes it easier for me to pinpoint the root cause.

Based on the model configurations you provided and the way you compiled MOM6SIS2, I have the following recommendations:

1. The error message you encountered, FATAL from PE 0: NETCDF ERROR: NetCDF: Variable not found File=INPUT/atmos_mosaic_tile1Xland_mosaic_tile1.ncField=ncells is mainly due to the fact that the FMS version in MOM6-examples is slightly out-of-date (2019.01), but the format of your atmos_mosaic_tile1Xland_mosaic_tile1.nc file is created for a later version of FMS (e.g., 2022.04). To resolve this, you can use the following command (assume you have nco tool in your system):

ncks --fix_rec_dmn ncells atmos_mosaic_tile1Xland_mosaic_tile1.nc atmos_mosaic_tile1Xland_mosaic_tile1_new.nc

This command will convert the ncells dimension from unlimited to fixed dimension so that the old FMS can recognize it. Additionally, I noticed that you are missing SIS2 configuration files. If you'd like, you can grab the modifications for your model configurations from here:

wget ftp.gfdl.noaa.gov:/pub/Yi-cheng.Teng/Archive_mod.tar.gz

2. Since you're planning to use MOM6 with OBCs, categorized as a 'regional-MOM6' application, I recommend utilizing the source codes from CEFI-regional-MOM6, instead of those from MOM6-examples This repository offers the most up-to-date source codes and model configurations tailored specifically for MOM6 regional applications. The model build process is also relatively straightforward. You can either refer to the instructions here, or simply follow the steps below:

First, clone the CEFI-regional-MOM6 repository with submodules:

git clone https://github.com/NOAA-GFDL/CEFI-regional-MOM6.git --recursive

Next, create folder for your machine configuration files (assume you already have it. In your case it isposeidon-intel.mk):

cd CEFI-regional-MOM6/builds
mkdir poseidon
cp /THE_PATH_OF YOUR_MKFILE/poseidon-intel.mk ./poseidon/intel.mk
touch  ./poseidon/intel.env

Then, build MOM6SIS2:

#build mom6sis2 on poseidon with intel compiler in production mode; you can also do  `-t debug` for debug mode
./linux-build.bash -m poseidon -p intel -t prod -f mom6sis2 

If the build completes successfully, you should find the executable here:
build/poseidon-intel/shared/prod/MOM6SIS2

Let me know if anything is unclear, I'm here to help. CC @charliestock to keep you in the loop.

[zi-hua-liu]

@yichengt900, thank you very much for your assistance! The solutions you provided were extremely helpful. I have implemented both, and they effectively resolved the issue with the fatal error message. Also thanks for letting me know that the version I used is old.

• For the old verison that I use, there are other issues came up ( i.e. 'FATAL from PE 0: xgrid_mod: tile is not in grid1%tile(:)' )... This issue may be attributed to incompatibilities with other files when using the old version. Given that the version I am currently using is outdated, I decided not to pursue this problem further, and have upgraded to the most up-to-date CEFI-regional-MOM6 as suggested.

• For CEFI-regional-MOM6, following your guidance, I successfully compiled the mom6-sis2 coupled mode without any issues. I am able to generate both the production (mom6sis2-prod) and debug (mom6sis2-debug) modes. 'mom6sis2-prod' has been tested with your modified model configuration (also thanks for the modifications and providing 'SIS' related files), and it works very well. I also modified my input files ( due to changes on domain size and horizontal coordinates ), and executed the updated regional mom6 model with this new setup. This adjustment also proceeded without any issues.

However, I am encountering new issues when modified 'MOM_input' file, and would greatly appreciate any insights:

1. I try to align the 'MOM_input' parameters closely with those used in the global model, incorporating both parameters used in global model and the obcs parameters, mostly only changing MOM_input file. Then, the model crashes again during initialization, with error like 'FATAL from PE 0: open_netcdf_file: File INPUT/Invalid argument'. This error occurs with both your modified configuration and my updated version. To aid in diagnosing the problem, I've shared my updated model configuration here.
   In the shared file, there are two MOM_input files. The model runs smoothly using 'MOM_input_org', however, when I switch to 'MOM_input_xiao_obcs', it results in the above fatal error. I have tried to solve this problem but failed... Any insights into the causes would be greatly appreciated.

2. Although I have the debug mode of the regional mom6-sis2 coupled version. I am not sure how to use that. Specifically, I'm wondering if I need to configure any output parameters to aid in debugging. Are there any available examples or guides that I could refer to for this purpose?

Thank you very much for your assistance. I apologize for the barrage of questions...

[yichengt900]

@zi-hua-liu, I took a quick look at your MOM_input. Below are some corrections to some of the runtime parameters:

1. NIGLOBAL = 77
2. NJGLOBAL = 122
3. We are not currently using USE_IDEAL_AGE_TRACER, so you can turn it off.
4. Rename TIDE_USE_SAL_SCALAR to SAL_SCALAR_APPROX.
5. Since you do not have tide_amp.nc at the moment, you will need to turn off INT_TIDE_DISSIPATION as well.
6. It seems you do not have the chl nc file (and you are not running with the BGC model at the moment), so change VAR_PEN_SW = False and PEN_SW_NBANDS = 1.

You should be able to run the new version with these fixes. Although I recommend you to have your chl and tide_amp files.

Regarding the debug mode, it currently involves compiling the code with additional flags (e.g., -g) to help you find the places where the model crashes more easily.

Feel free to let me know if you have any further questions.

[zi-hua-liu]

@yichengt900, super thanks! My model can run with no problem now. There are very helpful! (PS: Sorry for the fundamental mistakes on NI/NJ-GLOBAL... noticed that I did not send the updated version.) I will also add chl and tide_amp files later on. Thanks for the suggestions.