1. Added Carl's fix to mpas_io.F to use PIO 1.9x to build.

[supreethms1809]

      use netcdf_nf_data
   #endif
2. Added -r8 -O3 -byteswapio -Mfree -m64 -Mnofma -acc -ta=tesla:cc60 -Minfo=accel flag set to compile the data transfer openacc code in mpas_pool_routines.F

The title above should be a 1 line short summary of the pull request (i.e. what the project the PR represents is intended to do).

Enter a description of this PR. This should include why this PR was created, and what it does.

Testing and relations to other Pull Requests should be added as subsequent comments.

See the below examples for more information.
MPAS-Dev/MPAS#930
MPAS-Dev/MPAS#931

[mgduda]

I don't think these changes should be needed in the atmosphere/gpu branch, though it may be that I've compiled the netCDF library in a different way from what @cponder has used.

@cponder Can you confirm that you're using the latest atmosphere/gpu branch when you see build failures with PIO 1.x?

@supreethms1809 In any case, I think @raghuraj19 is the owner of the "official" atmosphere/gpu branch, so in future we'd probably need to make PRs to his fork rather than to mine.

[cponder]

I'll get a fresh snapshot and then double-check. But I want Pranay's gpu_divergence_3d fixes in the branch so I can deal with his issue at the same time.

[mgduda]

@cponder @Pranay-Reddy-Kommera What are the gpu_divergence_3d fixes, exactly?

[Pranay-Reddy-Kommera]

Dear Michael Duda, For the GPU implementation, we have added few additional variables and calls to certain subroutines. In newer (5.2) MPAS version, "divergence_3d" variable was removed from computations but we did not remove it in GPU implementation. This was resulting in "segmentation fault". This bug is from our side, we are currently comparing old (4.x) and newer versions of MPAS. Once we have a complete understanding of these additional variables which we need to remove, we will make changes to the code and make sure it is producing the expected output, and then submit it to your repository. Thanks, pranay

…

On Wed, Oct 18, 2017 at 4:44 PM, Michael Duda ***@***.***> wrote: @cponder <https://github.com/cponder> @Pranay-Reddy-Kommera <https://github.com/pranay-reddy-kommera> What are the gpu_divergence_3d fixes, exactly? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APGmNudZaKV6qym0-yvW4av8h2sV77BJks5stn9HgaJpZM4P43kU> .

-- With Regards, Pranay Reddy Kommera Graduate Student, Dept. of Electrical and Computer Engineering, University of Wyoming, pranay.kommera@gmail.com, +1-307-399-3046

[mgduda]

@Pranay-Reddy-Kommera Thanks very much for the explanation. If you all need any assistance in understanding the changes to the 3-d divergence damping between v4.0 and v5.2, just let me know.

[cponder]

*On 10/18/2017 02:50 PM, Michael Duda wrote:*

I don't think these changes should be needed in the |atmosphere/gpu| branch, though it may be that I've compiled the netCDF library in a different way from what @cponder <https://github.com/cponder> has used. @cponder <https://github.com/cponder> Can you confirm that you're using the latest |atmosphere/gpu| branch when you see build failures with PIO 1.x? @supreethms1809 <https://github.com/supreethms1809> In any case, I think @raghuraj19 <https://github.com/raghuraj19> is the owner of the "official" |atmosphere/gpu| branch, so in future we'd probably need to make PRs to his fork rather than to mine.

Hey -- this *did* build okay, using the atmosphere/gpu snapshot I just took today (2017-10-21). I don't see any changes in the file src/framework/mpas_io.F from the 2017-09-27 or 2017-10-18 snapshots of the CPU code. Do you know what fixed the problem? It looks like a PIO2 change had been added to the atmosphere/gpu branch 282    else if (ioformat == MPAS_IO_NETCDF4) then else if (ioformat == MPAS_IO_NETCDF4) then 283       pio_iotype = PIO_iotype_netcdf4p                          pio_iotype = PIO_iotype_netcdf4p 284 >    #ifdef USE_PIO2 285                                                   >              pio_mode = 0 286 >    #else 287       pio_mode = PIO_64BIT_OFFSET           pio_mode = PIO_64BIT_OFFSET 288                                                   >   #endif 289    end if end if but then removed more recently. Thanks,                   Carl

…

------------------------------------------------------------------------ *On 10/10/2017 02:49 PM, Carl Ponder wrote: * Ok here's what I have: I'm using this software-stack module purge                                    # Get rid of residual environment. module load PrgEnv/PGI+OpenMPI/2017-10-06       # Based on pgi/17.9 module load cuda module load pgi module load openmpi module load pnetcdf module load netcdf-f module load netcdf-c *module load pio/1.10.0* to build & run on the hsw_p100 nodes of the PSG cluster. (Note that I'm backing-down the PIO level to the 1.* branch) Both MPAS-A & MPAS-O are working CPU-only. I'd gotten the MPAS-A codebase by running the command git clone https://github.com/MPAS-Dev/MPAS.git on 2017-09-27. The codebase I get using this command git clone https://github.com/raghuraj19/MPAS.git doesn't build, I get stuck here mpif90 -O3 -o build_tables build_tables.o atmphys_build_tables_thompson.o -L../../framework -L../physics -lphys -lframework build_tables.o: In function `.C1_283': build_tables.F:(.data+0xd8): undefined reference to `pio_support_' build_tables.F:(.data+0xe8): undefined reference to `pio_types_' build_tables.F:(.data+0xf0): undefined reference to `pio_msg_mod_' build_tables.F:(.data+0xf8): undefined reference to `esmf_basemod_'                  ...... where it's missing linkage-information. The CPU-only codebase is passing more paths to this part of the build mpif90 -O3 -o build_tables build_tables.o atmphys_build_tables_thompson.o -L../../framework -L../physics -lphys -lframework -L/cm/extra/apps/PIO/1.10.0/PGI-17.9_CUDA-9.0.176_OpenMPI-1.10.7_HDF5-1.10.1_NetCDF-C-4.4.1.1_NetCDF-F-4.4.4/lib -lpio -L/cm/extra/apps/PNetCDF/1.8.1/PGI-17.9_OpenMPI-1.10.7_CUDA-9.0.176/lib -lpnetcdf -L/cm/extra/apps/NetCDF-C/4.4.1.1/PGI-17.9_PNetCDF-1.8.1_HDF5-1.10.1_OpenMPI-1.10.7_CUDA-9.0.176/lib -L/cm/extra/apps/NetCDF-F/4.4.4/PGI-17.9_OpenMPI-1.10.7_CUDA-9.0.176/lib -lnetcdff -lnetcdf -L../../external/esmf_time_f90 -lesmf_time so it looks like some changes need to be migrated over. For the CPU-only build, I had to add these lines to the source-file src/framework/mpas_io.F  8 module mpas_io  9 10    use mpas_derived_types 11    use mpas_attlist 12    use mpas_dmpar 13    use mpas_log 14 *15 #ifndef USE_PIO2* *16    use netcdf_nf_data* *17 #endif* 18    use pio 19    use piolib_mod 20    use pionfatt_mod 21    use pio_types without which, I would get these errors in the build: PGF90-S-0087-Non-constant expression where constant expression required (mpas_io.F: 30) PGF90-S-0155-Intrinsic not supported in initialization: achar (mpas_io.F: 31) PGF90-S-0087-Non-constant expression where constant expression required (mpas_io.F: 44)   0 inform,   0 warnings,   3 severes, 0 fatal for mpas_io PGF90-S-0038-Symbol, nf_fill_char, has not been explicitly declared (mpas_io.F)   0 inform,   0 warnings,   1 severes, 0 fatal for mpas_io_init PGF90-S-0038-Symbol, nf_fill_char, has not been explicitly declared (mpas_io.F)   0 inform,   0 warnings,   1 severes, 0 fatal for mpas_io_set_iotype                     ..... What I don't get here, is why the MPAS-A 5.1 source didn't need this fix, the corresponding file content is this  8 module mpas_io  9 10    use mpas_derived_types 11    use mpas_attlist 12    use mpas_dmpar 13    use mpas_io_units 14    use mpas_abort, only : mpas_dmpar_global_abort 15 16    use pio 17    use piolib_mod 18    use pionfatt_mod 19    use pio_types so I'm wondering if there's a cleaner fix than the one I have above.             -- Carl

----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------