NetCDF 4.9.0: undefined reference to `ncp_win32_api' on MinGW (libncpoco)

[Alexander-Barth]

I tried to update the NetCDF version available to julia users.
However, netCDF 4.9.0 fails to build during cross-compilation for x86_64-w64-mingw32 with the error "undefined reference to `ncp_win32_api'" .

• the version of the software with which you are encountering an issue

NetCDF 4.9.0

• environmental information (i.e. Operating System, compiler info, java version, python version, etc.)

target x86_64-w64-mingw32 (cross compiled from Linux)
With gcc version: x86_64-w64-mingw32-gcc (GCC) 4.8.5

• a description of the issue with the steps needed to reproduce it

Here are the steps how we compile NetCDF (https://github.com/JuliaPackaging/Yggdrasil/blob/master/N/NetCDF/common.jl#L59)

./configure --prefix=/workspace/destdir --build=x86_64-linux-musl --host=x86_64-w64-mingw32 --enable-shared --disable-static --disable-dap-remote-tests --disable-testsets
make LDFLAGS="-L/workspace/destdir/bin -lhdf5-0 -lhdf5_hl-0 -lcurl-4 -lz -no-undefined -Wl,--export-all-symbols" -j8

The build step fails with:

[22:59:58] make[2]: Entering directory '/workspace/srcdir/netcdf-c-4.9.0/liblib'           
[22:59:58] /bin/sh ../libtool  --tag=CC   --mode=compile cc -DHAVE_CONFIG_H -I. -I..  -I../include -I../include -I../oc2 -I../libnczarr  -I../libhdf5  -I../libdap2 -I
../oc -I../libdap4 -I../libncxml -I../libsrc4 -I../libnczarr -I../libncpoco -I/workspace/destdir/include  -std=c99 -fno-strict-aliasing -MT libnetcdf_la-nc_initialize
.lo -MD -MP -MF .deps/libnetcdf_la-nc_initialize.Tpo -c -o libnetcdf_la-nc_initialize.lo `test -f 'nc_initialize.c' || echo './'`nc_initialize.c
[22:59:58] libtool: compile:  cc -DHAVE_CONFIG_H -I. -I.. -I../include -I../include -I../oc2 -I../libnczarr -I../libhdf5 -I../libdap2 -I../oc -I../libdap4 -I../libncx
ml -I../libsrc4 -I../libnczarr -I../libncpoco -I/workspace/destdir/include -std=c99 -fno-strict-aliasing -MT libnetcdf_la-nc_initialize.lo -MD -MP -MF .deps/libnetcdf
_la-nc_initialize.Tpo -c nc_initialize.c  -DDLL_EXPORT -DPIC -o .libs/libnetcdf_la-nc_initialize.o
[22:59:58] mv -f .deps/libnetcdf_la-nc_initialize.Tpo .deps/libnetcdf_la-nc_initialize.Plo                                                     
[22:59:58] /bin/sh ../libtool  --tag=CC   --mode=link cc  -std=c99 -fno-strict-aliasing -version-info 20:0:1  -L/workspace/destdir/bin -lhdf5-0 -lhdf5_hl-0 -lcurl-4 -
lz -no-undefined -Wl,--export-all-symbols -o libnetcdf.la -rpath /workspace/destdir/lib libnetcdf_la-nc_initialize.lo ../libdispatch/libnetcdf2.la ../libdispatch/libd
ispatch.la ../libsrc/libnetcdf3.la  ../libhdf5/libnchdf5.la  ../libdap2/libdap2.la ../oc2/liboc.la ../libdap4/libdap4.la ../libncxml/libncxml.la ../libsrc4/libnetcdf4
.la ../libnczarr/libnczarr.la  ../libncpoco/libncpoco.la -lm -lsz -lxml2 -lhdf5-0 -lhdf5_hl-0 -lcurl-4 -lz                             
[22:59:58] libtool: link: cc -shared  .libs/libnetcdf_la-nc_initialize.o  -Wl,--whole-archive ../libdispatch/.libs/libnetcdf2.a ../libdispatch/.libs/libdispatch.a ../
libsrc/.libs/libnetcdf3.a ../libhdf5/.libs/libnchdf5.a ../libdap2/.libs/libdap2.a ../oc2/.libs/liboc.a ../libdap4/.libs/libdap4.a ../libncxml/.libs/libncxml.a ../libs
rc4/.libs/libnetcdf4.a ../libnczarr/.libs/libnczarr.a ../libncpoco/.libs/libncpoco.a -Wl,--no-whole-archive  -L/workspace/destdir/bin -lsz -lxml2 -lhdf5-0 -lhdf5_hl-0
 -lcurl-4 -lz  -Wl,--export-all-symbols   -o .libs/libnetcdf-19.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libnetcdf.dll.a                 
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x40): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x4e): undefined reference to `ncp_win32_api'                                                     
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x5c): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x6a): undefined reference to `ncp_win32_api'                                                     
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x78): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x86): more undefined references to `ncp_win32_api' follow                                        
[22:59:58] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/4.8.5/../../../../x86_64-w64-mingw32/bin/ld: ../libncpoco/.libs/libncpoco.a(ncpoco.o): bad reloc 
address 0x0 in section `.pdata'                                                                                                                                       
[22:59:58] collect2: error: ld returned 1 exit status
[22:59:58] make[2]: *** [Makefile:612: libnetcdf.la] Error 1                                                                                                          
[22:59:58] make[2]: Leaving directory '/workspace/srcdir/netcdf-c-4.9.0/liblib'
[22:59:58] make[1]: *** [Makefile:764: all-recursive] Error 1                                                                                                         
[22:59:58] make[1]: Leaving directory '/workspace/srcdir/netcdf-c-4.9.0'                                                                                              
[22:59:58] make: *** [Makefile:609: all] Error 2                                                                   
[22:59:58]  ---> make LDFLAGS="${LDFLAGS_MAKE}" -j${nproc}                                                                                                            
[22:59:58]  ---> make LDFLAGS="${LDFLAGS_MAKE}" -j${nproc}                                                                       
[22:59:58] Previous command exited with 2                                                                                                                             
[22:59:58] Child Process exited, exit code 2              
┌ Warning: Build failed, the following log files were generated:                                  
│   - ${WORKSPACE}/srcdir/netcdf-c-4.9.0/config.log                                        

Attached is the file config.log

Thank you for your help.

[Alexander-Barth]

Maybe --disable-plugins is necessary for Windows?

netcdf-c/.github/workflows/run_tests_win_mingw.yml

Line 38 in 5df5539

run: ./configure --enable-hdf5 --enable-dap --disable-dap-remote-tests --disable-static --disable-plugins --disable-byterange --disable-dap-remote-tests --disable-logging

(the compilation succeeds with this option)

[DennisHeimbigner]

It looks like the file libncpoco/cp_win32.c is not being compiled.
Can you check your build to see if it compiles that file?

[Alexander-Barth]

Indeed, this file is not compiled:

$ LANG=C ls -la * .libs/
-rwxr-xr-x 1 abarth abarth   278 Jun  10 23:04 CMakeLists.txt
-rwxr-xr-x 1 abarth abarth  2043 Jun  10 23:04 COPYRIGHT
-rw-r--r-- 1 abarth abarth 23996 Jun  13 09:19 Makefile
-rwxr-xr-x 1 abarth abarth  1199 Jun  10 23:04 Makefile.am
-rw-rw-r-- 1 abarth abarth 25603 Jun  10 23:04 Makefile.in
-rw-rw-r-- 1 abarth abarth  4617 Jun  10 23:04 README.md
-rw-rw-r-- 1 abarth abarth  1401 Jun  10 23:04 SourceLicence
-rwxr-xr-x 1 abarth abarth  5354 Jun  10 23:04 cp_test.c
-rwxr-xr-x 1 abarth abarth  3689 Jun  10 23:04 cp_unix.c
-rw-r--r-- 1 abarth abarth   266 Jun  13 09:13 cp_unix.lo
-rwxr-xr-x 1 abarth abarth  3733 Jun  10 23:04 cp_win32.c
-rwxr-xr-x 1 abarth abarth   100 Jun  10 23:04 cptestlib.c
-rw-r--r-- 1 abarth abarth   923 Jun  13 09:13 libncpoco.la
-rwxr-xr-x 1 abarth abarth  2748 Jun  10 23:04 ncpoco.c
-rwxr-xr-x 1 abarth abarth  2327 Jun  10 23:04 ncpoco.h
-rw-r--r-- 1 abarth abarth   264 Jun  13 09:13 ncpoco.lo

.libs/:
total 24
drwxr-xr-x 2 abarth abarth 4096 Jun  13 09:13 .
drwxrwxr-x 4 abarth abarth 4096 Jun  13 09:19 ..
-rw-r--r-- 1 abarth abarth 2620 Jun  13 09:13 cp_unix.o
-rw-r--r-- 1 abarth abarth 5100 Jun  13 09:13 libncpoco.a
lrwxrwxrwx 1 abarth abarth   15 Jun  13 09:13 libncpoco.la -> ../libncpoco.la
-rw-r--r-- 1 abarth abarth 2092 Jun  13 09:13 ncpoco.o

Maybe the build system is confused since we do a cross-compiliation (as cp_unix.o is present).

[DennisHeimbigner]

Ok, the problem is probably the conditional code in libncpoco/Makefile.am.
It currently read as follows:

if ! ISMSVC 
if ! ISMINGW 
libncpoco_la_SOURCES += cp_unix.c
else
libncpoco_la_SOURCES += cp_win32.c
endif
else
libncpoco_la_SOURCES += cp_win32.c
endif

Btw, the value of ISMSVC and ISMINGW comes from this code in configure.ac

case "`uname`" in
  CYGWIN*) ISCYGWIN=yes;;
  Darwin*) ISOSX=yes;;
  WIN*) ISMSVC=yes;;
  MINGW*) ISMINGW=yes;;
esac

For some reason, this is not working. Does anything stand out to you?

[DennisHeimbigner]

I believe you are correct that the cross compile is the problem. Not sure how to
fix the above tests to get this correct.

[Alexander-Barth]

Inside the docker container from the Julia build system, I run these tests on a bash shell:

sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $target  # probably julia specific
x86_64-w64-mingw32
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # uname -a
MSYS_NT-6.3 gher17 2.8.2(0.313/5/3) Wed Feb 13 00:49:00 UTC 2013 x86_64 Cygwin
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # case "`uname`" in
  CYGWIN*) ISCYGWIN=yes;;
  Darwin*) ISOSX=yes;;
  WIN*) ISMSVC=yes;;
  MINGW*) ISMINGW=yes;;
esac
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISCYGWIN

sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISOSX

sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISMSVC

sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISMINGW

sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # uname
MSYS_NT-6.3

Maybe we need to test: MSYS*) ISMINGW=yes;; Or there could be a source file e.g. cp_platform.c which would conditionally include cp_unix.c or cp_win32.c

#ifdef _WIN32
#include "cp_win32.c"
#else
#include "cp_unix.c"
#endif

[WardF]

@Alexander-Barth Out of curiosity, is the Docker container you mentioned one that could be run/used outside of the Julia packaging system? I'm digging into the README.md and want to make sure I don't dive too deep into this without knowing if it's leading anywhere. But the idea of a Docker image can generate cross-platform netCDF packages is interesting to me, and I'd love to take a look at what you're using, if it is possible to in a stand-alone fashion. FWIW, I have a good bit of familiarity with Docker. Thanks!

[Alexander-Barth]

@WardF The build environment (https://binarybuilder.org/) is in fact a julia package which setups the container and the compiler toolchain and it uses the docker image from alpine linux as a base. While on Windows and Mac OS, the docker engine is also used, it seem that on Linux a more lightweight and custom sandbox runtime is used.

There is more technical information here: https://docs.binarybuilder.org/dev/rootfs/

The resulting binaries (libraries, executable...) are in fact independent of julia and are just simple tarballs (one per architecture / OS ).

I don't know too much about the internals (and maybe @staticfloat can correct me if I misunderstood something), but as a user a lot of things (handling the build chain, injecting software dependencies in the container) happen automatically. If a build fails with, the --debug option, allows one to automatically drops into a shell. Per default BinaryBuilder also keeps track (log files, sources) of previous attempts. For new packages, there is also a command line Wizard were you can specify your dependencies and supported platforms. The main entry point as a developer are the build_tarballs.jl julia script which essentially contain a shell script to build the binaries, e.g.

https://github.com/JuliaPackaging/Yggdrasil/blob/master/U/UDUNITS/build_tarballs.jl

Resulting binaries:

https://github.com/JuliaBinaryWrappers/UDUNITS_jll.jl/releases/tag/UDUNITS-v2.2.28%2B0

For NetCDF, it is a bit more complicated as there are shared dependencies (libcurl among others) that need to be synchronized between Julia and NetCDF:

https://github.com/JuliaPackaging/Yggdrasil/tree/master/N/NetCDF
https://github.com/JuliaBinaryWrappers/NetCDF_jll.jl/releases/tag/NetCDF-v400.802.103%2B0

(sadly the platform support of NetCDF is not nearly as large as UDUNITS, but this is mostly due to the fact that HDF5 does not support cross-compilation and some Windows issues)

[WardF]

Interesting, thank you! That is worth reading through and educating myself; I am not super familiar with Julia, but I see the opportunity to extract chunks of technologies I am much more familiar with. If nothing else, it is a really interesting toolchain to parse, and I quite like the idea of being able to mint cross-platform libraries using docker. Thanks for the comprehensive introduction!

[DWesl]

Btw, the value of ISMSVC and ISMINGW comes from this code in configure.ac

netcdf-c/configure.ac

Lines 105 to 111 in 26c5582

	case "`uname`" in
	CYGWIN*) ISCYGWIN=yes;;
	Darwin*) ISOSX=yes;;
	WIN*) ISMSVC=yes;;
	MINGW*) ISMINGW=yes;;
	MSYS*) ISMINGW=yes;;
	esac

For some reason, this is not working. Does anything stand out to you?

This seems to be the kind of thing the --host option to ./configure is used for; using AC_CANONICAL_HOST gets a standard value, there are suggestions for using that value here, which suggest something like this:

AC_CANONICAL_HOST
AS_CASE([$host],
    [*-*-cygwin], [ISCYGWIN=yes],
    [*-*-darwin*], [ISOSX=yes],
    [*-*-mingw*], [ISMINGW=yes],
    [*-*-msys], [ISMINGW=yes],
    [*-*-win*], [ISMSVC=yes],
    []
)

using a macro from here and the guesses from here. For some reason, GNU Autoconf doesn't seem to have code for guessing MicroSoft Visual C, so compiling using MSVC and the Autotools build scripts (./configure && make) will require specifying --build and/or --host or some fallback code to catch the failure to guess the build system and set the variables as in the current code.

(I think one of automake or libtool already pulls in AC_CANONICAL_HOST, so this wouldn't add new code. --host and friends will recognize winnt, windows, wince, and certain variations of those if specified on the command line, configure just won't guess those on its own.)

I made this change on my fork, and have seen no new failures on Cygwin or MinGW.