PIO segfaults on melvin

[jgfouca]

Test: ERS.f45_g37.B1850C5
Platform: gcc/5.1.0, openmpi/1.8.7, netcdf/4.3.2, pnetcdf/1.5.0, netcdf-fortran/4.4.1

Seg fault:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x7F5D8FB1BB77
#0 0x7F38B6AA5B77
#1 0x7F38B6AA4D70
#2 0x35D703269F
#3 0x7F38B829B36C
#4 0x7F38B827C14E
#5 0x7F38B8273A70
#1 0x7F5D8FB1AD70
#2 0x35D703269F
#3 0x7F5D9131136C
#4 0x7F5D912F214E
#5 0x7F5D912E9A70
#6 0x12A213E in nfmpi_put_vara_int_all_
#6 0x12A213E in nfmpi_put_vara_int_all_
#7 0x11DFB98 in __pionfput_mod_MOD_put_vara_1d_int
#7 0x11DFB98 in __pionfput_mod_MOD_put_vara_1d_int
#8 0x11EEA50 in __pionfput_mod_MOD_put_var_1d_int
#8 0x11EEA50 in __pionfput_mod_MOD_put_var_1d_int
#9 0x11EEB2E in __pionfput_mod_MOD_put_var_vdesc_1d_int
#9 0x11EEB2E in __pionfput_mod_MOD_put_var_vdesc_1d_int
#10 0x4A288A in __cam_history_MOD_wshist
#10 0x4A288A in __cam_history_MOD_wshist
#11 0x4ABBDF in __cam_history_MOD_write_restart_history
#11 0x4ABBDF in __cam_history_MOD_write_restart_history
#12 0x4BCAD9 in __cam_restart_MOD_cam_write_restart
#12 0x4BCAD9 in __cam_restart_MOD_cam_write_restart
#13 0x48EA0F in __cam_comp_MOD_cam_run4
#13 0x48EA0F in __cam_comp_MOD_cam_run4
#14 0x488A1B in __atm_comp_mct_MOD_atm_run_mct
#14 0x488A1B in __atm_comp_mct_MOD_atm_run_mct
#15 0x428059 in __component_mod_MOD_component_run
#15 0x428059 in __component_mod_MOD_component_run
#16 0x41705A in __cesm_comp_mod_MOD_cesm_run
#16 0x41705A in __cesm_comp_mod_MOD_cesm_run

[jayeshkrishna]

I am able to reproduce this issue on my laptop (Ubuntu 14.04 + 64-bit + gnu 4.8.4 + mpich 1.5 + pnetcdf 1.6.1). The code segfaults in pnetcdf. The missing stack frames (not listed above) is shown below (from my laptop),

#3 0x466A9A in ncmpii_in_swapn at convert_swap.m4:185 (discriminator 2)
#3 0x466A9A in ncmpii_in_swapn at convert_swap.m4:185 (discriminator 2)
#4 0x439EF5 in ncmpii_getput_vars at getput.m4:863
#5 0x431B51 in ncmpi_put_vara_int_all at getput.m4:357 (discriminator 2)
#4 0x439EF5 in ncmpii_getput_vars at getput.m4:863
#6 0x40528E in nfmpi_put_vara_int_all_ at put_vara_int_allf.c:57

[jayeshkrishna]

Since reproducing the issue takes about 3 hours (running the test until it crashes) I finally resorted to writing out PIO and pnetcdf test cases that can potentially reproduce the problem.
I am now able to reproduce the issue with a simple PIO test case that performs a put of an integer parameter and also using a pnetcdf test case that uses the underlying pnetcdf API used by PIO.

[jayeshkrishna]

After consulting with Rob Latham (MPI I/O+ PnetCDF@ANL) and Wei Keng (PnetCDF @ northwestern) the problem seems to be a performance optimization (in place swapping to save memory) inside PnetCDF.
After turning off this optimization in PnetCDF the simple test cases work for me (no longer crashes). I am currently trying out the ACME case on my laptop and will soon update the results.

[rljacob]

Great work! Might be faster to tell @jgfouca how to try it.

[jayeshkrishna]

The PnetCDF performance optimization can be turned off by rebuilding PnetCDF with "--disable-in-place-swap" (./configure ... --disable-in-place-swap) and then rebuilding the case.

[jayeshkrishna]

The ERS.f45_g37.B1850C5 runs successfully on my laptop after rebuilding the case with a version of PnetCDF with "in place swap" turned off (./configure ... --disable-in-place-swap).

[jgfouca]

@jayeshkrishna , I don't suppose there's a code change to PIO that could get around this?

[rljacob]

What is the F-case doing to expose this bug?

[jayeshkrishna]

The case (CAM history code) does a PIO put with a temp array created from an integer parameter. My assumption is that the gnu compiler does aggressive optimization and retains the "parameter status" of the variable while other compilers don't.
PIO puts with integers that are not parameters (integer, parameter) work fine on all machines+compilers.
Since this issue arises only with gcc on Linux I am not sure if it is worth our time to try "fixing" it in PIO (Note that none of this code will be used once we move to PIO2 - different calls, different language).

[jgfouca]

@jayeshkrishna , if the fix is hard, then don't worry about it. My concern is that, with the special configuration needed for pnetcdf, Melvin will no longer be able to use the supported pnetcdf, so we'll have to maintain our own pnetcdf build. Annoying, but hardly end of the world.

[jayeshkrishna]

Actually apart from copying the integer parameters to a regular integer locally before a put (very hacky) I don't know of any other solutions right now. Creating these temp arrays (as in the current CAM history code) should have been the more elegant way if it worked (unfortunately it does not).

So I am not sure of a "clean way" of fixing this issue right now by modifying the source code. And as I mentioned above this issue will go away once we move to PIO2.

[jayeshkrishna]

@jgfouca / @rljacob : Do we close this PR (Although Jim was not able to successfully run the test on Melvin after turning the PnetCDF optimizations off, he has now switched to NetCDF on Melvin for these tests. PIO+NetCDF does not exhibit the same issue and he is now able to run the tests successfully on Melvin)?

[jgfouca]

@jayeshkrishna , i have a viable workaround and, as you've said, PIO is going away, so I will close.

[jayeshkrishna]

PIO1 (not PIO) is going away, hopefully soon :)