CI: configure script's m4/* library checks begin with CFLAGS=""

[jimklimov]

The goal of emptying the CFLAGS is to determine (from pkg-config, CLI args or other sources) the actual set of tokens needed by a dependency.

This however leads to NOT using some earlier discovered (or caller-provided) CFLAGS in some of those tests, notably with hints about customized bitness or target architecture. As a result, a build system capable of building for several targets actually checks some of the dependencies for the compiler-default target. Most of the time this is okay if dependencies are all present consistently for all targets, but fails if targets are provisioned with different abilities.

As a practical example, OpenIndiana is capable of 32-bit (i386) and 64-bit (amd64) builds. It provides a libgd which is unfortunately broken for 32-bit version (packaging no longer delivers some shared objects used by libgd.so for the platform, but the library itself, headers and pkg-config metadata are delivered). Tests with gcc-7 compiler (targeting 32-bit by default) fail to find a functional libgd even if we call CFLAGS=-m64 ./configure --with-cgi for an explicit 64-bit build. Some code is tested for 64-bit compile-/link-ability, but m4/nut_check_libgd.m4 is subject to compiler defaults, tests the broken 32-bit library, and bails out.

It also does not help when pkg-config returns linkage info correct for the requested target platform (per --with-pkg-config-path settings), but the compiler tries to build for another:

configure:23778: checking for Mozilla NSS version via pkg-config
configure:23784: result: 3.98 found
configure:23811: checking for Mozilla NSS cflags
configure:23829: result: -I/usr/include/mps
configure:23832: checking for Mozilla NSS ldflags
configure:23850: result: -L/usr/lib/mps/amd64 -R/usr/lib/mps/amd64 -lnss3 -lsmime3 -lssl3 -lnssutil3 -R/usr/lib/mps/amd64 -lplds4 -lplc4 -lnspr4 -lpthread -ldl -lposix4
configure:23856: checking for NSS_Init
configure:23856: /usr/lib/ccache/gcc-7 -o conftest -I/usr/include/mps  -I/export/home/abuild/nut/tmp/include -L/export/home/abuild/nut/tmp/lib  conftest.c -L/usr/lib/mps/amd64 -R/usr/lib/mps/amd64 -lnss3 -lsmime3 -lssl3 -lnssutil3 -R/usr/lib/mps/amd64 -lplds4 -lplc4 -lnspr4 -lpthread -ldl -lposix4 >&5
ld: fatal: file /usr/lib/mps/amd64/libnss3.so: wrong ELF class: ELFCLASS64
ld: fatal: file /usr/lib/mps/amd64/libsmime3.so: wrong ELF class: ELFCLASS64
ld: fatal: file /usr/lib/mps/amd64/libssl3.so: wrong ELF class: ELFCLASS64
ld: fatal: file /usr/lib/mps/amd64/libnssutil3.so: wrong ELF class: ELFCLASS64
ld: fatal: file /usr/lib/mps/amd64/libplds4.so: wrong ELF class: ELFCLASS64
ld: fatal: file /usr/lib/mps/amd64/libplc4.so: wrong ELF class: ELFCLASS64
ld: fatal: file /usr/lib/mps/amd64/libnspr4.so: wrong ELF class: ELFCLASS64
ld: fatal: file processing errors. No output written to conftest
collect2: error: ld returned 1 exit status
configure:23856: $? = 1

This issue is about thinking some way around the problem. For example, we could track COMPONENT_CFLAGS etc. in their dedicated variables right away, maybe kept private until success, and mix with the stored copy of original CFLAGS etc. for practical tests (perhaps also somehow determine which flags are "systemic")?.. An example prospective solution is in m4/nut_check_libnss.m4 in fact, where it honours original -m32/-m64 if present in the flags (and if GCC==yes), attaching them to the copy of $CC used by the test.

Also it would be wise to find a way of checking/ensuring that pkg-config results (or search path?) match the compiler default/requested target architecture, so configure-time discoveries are representative of what the actual build/link would achieve.

This would impact many of the m4 scripts, and possibly code inside the configure.ac script source.

[jimklimov]

As posted into PR #2675, I went with temporary variables to collect flags for the current dependency in each m4/nut_check_lib*.m4 file, combining them into a CFLAGS, LIBS etc. value used by the standard autoconf checks.

Note that AC_SEARCH_LIBS changes LIBS on its own, so we pick up the changes to remember in depLIBS afterwards, in the few files where it happens.