OpenDAP hang on Mingw-w64

[mjwoods]

• Windows 10 x64
• Current version of mingw-w64
  • Available from https://www.msys2.org/wiki/MSYS2-installation
• Install netcdf-4.7.4 package (e.g. pacman -S mingw64/mingw-w64-x86_64-netcdf)
• Try to access an OpenDAP dataset with ncdump
• For example, the following command hangs until interrupted by user:
  • ncdump -hs https://cida.usgs.gov/thredds/dodsC/stageiv_combined

The problem appears to be triggered by compiler optimisations. I have tested gcc-8, gcc-10 and clang-12, and the hang occurs for all of them with the default optimizations chosen by CMake. The hang does NOT occur with optimisation at -O1 or lower.

Looking more closely at the optimization options for gcc, I found a specific optimization that causes the problem: -foptimize-sibling-calls. OpenDAP appears to work correctly if the package is compiled with -O3 -fno-optimize-sibling-calls.

Although there seems to be a workaround, it would be better to understand the root cause of the problem, which is why I am reporting it here.

[mjwoods]

In case it helps, here is a backtrace from gdb after interrupting the hanging ncdump example given earlier. To trigger the hang, the package was built with -Og -foptimize-sibling-calls.

(gdb) run -hs  https://cida.usgs.gov/thredds/dodsC/stageiv_combined
Starting program: C:\msys64\mingw64\bin\ncdump.exe -hs  https://cida.usgs.gov/thredds/dodsC/stageiv_combined
...
Thread 1 (Thread 1272.0x2fb4):
#0  daplex (lvalp=lvalp@entry=0x1e121fe718, state=state@entry=0x13198e081e0) at C:/msys64/mingw64/x86_64-w64-mingw32/include/string.h:117
#1  0x00007ff8a7f4ce8e in dapparse (parsestate=parsestate@entry=0x13198e081e0) at dap.tab.c:1330
#2  0x00007ff8a7f4c76a in DAPparse (conn=conn@entry=0x13198c18370, tree=tree@entry=0x13198c16870, parsestring=<optimized out>) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/oc2/dapparse.c:502
#3  0x00007ff8a7f52a4e in ocfetch (state=state@entry=0x13198c18370, constraint=constraint@entry=0x0, kind=kind@entry=OCDDS, flags=flags@entry=0, rootp=<optimized out>, rootp@entry=0x1e121fe858) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/oc2/ocinternal.c:229
#4  0x00007ff8a7f49b5e in oc_fetch (link=link@entry=0x13198c18370, constraint=constraint@entry=0x0, dxdkind=dxdkind@entry=OCDDS, flags=flags@entry=0, rootp=rootp@entry=0x1e121fe928) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/oc2/oc.c:126
#5  0x00007ff8a7f59559 in dap_fetch (nccomm=nccomm@entry=0x13198c15c40, conn=0x13198c18370, ce=ce@entry=0x0, dxd=dxd@entry=OCDDS, rootp=rootp@entry=0x1e121fe928) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/libdap2/daputil.c:691
#6  0x00007ff8a7f5e601 in fetchpatternmetadata (dapcomm=dapcomm@entry=0x13198c15c40) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/libdap2/ncd2dispatch.c:2042
#7  0x00007ff8a7f623fa in NCD2_open (path=<optimized out>, mode=<optimized out>, basepe=<optimized out>, chunksizehintp=<optimized out>, mpidata=0x0, dispatch=0x7ff8a7fefe80 <NCD2_dispatch_base>, ncid=65536) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/libdap2/ncd2dispatch.c:426
#8  0x00007ff8a7ef33b6 in NC_open (path0=<optimized out>, omode=<optimized out>, omode@entry=0, basepe=basepe@entry=0, chunksizehintp=chunksizehintp@entry=0x0, useparallel=useparallel@entry=0, parameters=parameters@entry=0x0, ncidp=ncidp@entry=0x1e121feb1c) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/libdispatch/dfile.c:2117
#9  0x00007ff8a7ef34bb in nc_open (path=<optimized out>, omode=omode@entry=0, ncidp=ncidp@entry=0x1e121feb1c) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/libdispatch/dfile.c:657
#10 0x00007ff67c4259d2 in main (argc=1, argv=0x131973d4f80) at C:/msys64/home/user/MINGW-packages/mingw-w64-netcdf/src/netcdf-c-4.7.4/ncdump/ncdump.c:2383

[mjwoods]

Looking at the string.h file shown in the backtrace above, I noticed that line 117 is as follows:

115 #ifndef __CRT__NO_INLINE
116   __CRT_INLINE int __cdecl strncasecmp (const char *__sz1, const char *__sz2, size_t __sizeMaxCompare) { return _strnicmp (__sz1, __sz2, __sizeMaxCompare); }
117   __CRT_INLINE int __cdecl strcasecmp (const char *__sz1, const char *__sz2) { return _stricmp (__sz1, __sz2); }
118 #else
119 #define strncasecmp _strnicmp
120 #define strcasecmp _stricmp
121 #endif /* !__CRT__NO_INLINE */

The macro __CRT__NO_INLINE is normally undefined, so I tried another build with the compiler flags -Og -foptimize-sibling-calls -D__CRT__NO_INLINE. This also fixes the OpenDAP hang, although it would be good to understand why it works.

[DennisHeimbigner]

I assume it works because adding that -D flag
causes this code to be used:

#define strncasecmp _strnicmp
#define strcasecmp _stricmp

My guess is that there is some .h file that defines __CRT_INLINE
and that it should be included somewhere.

[mjwoods]

Hi Dennis,

Thanks for your comment. I suppose this could be an issue for the mingw-w64 team, but there may be something special about the way netcdf is using the string functions, so it seemed worthwhile to ask here first. Any clues you can offer will be helpful.

The __CRT_INLINE macro is defined in the file C:/msys64/mingw64/x86_64-w64-mingw32/include/_mingw.h, as follows:

#ifdef __cplusplus
# define __CRT_INLINE inline
#elif defined(_MSC_VER)
# define __CRT_INLINE __inline
#else
# if ((__MINGW_GNUC_PREREQ(4, 3) || defined(__clang__)) && __STDC_VERSION__ >= 199901L)
#  define __CRT_INLINE extern inline __attribute__((__gnu_inline__))
# else
#  define __CRT_INLINE extern __inline__
# endif
#endif

[DennisHeimbigner]

Well "strcasecmp" is conditionally defined in ncconfigure.h.
So if string.h is included after netcdf.h, then there might be a problem.

[mjwoods]

That was a good hint @DennisHeimbigner . I was able to patch netcdf-4.7.4 so that the redefinitions of strcasecmp were disabled under mingw-w64, and that does fix the hang. I'll try to submit a PR for the current master within the next week or so.