Build of sf on openSUSE fails: configure: error: libproj or sqlite3 not found in standard or given locations.

[Bob-O-Rama]

I am having difficulty getting sf to build in R under openSUSE Tumbleweed ( so latest "everything" so R 4.1.0, etc. after a zypper dup ) There were a couple dependency errors where the error itself was sort of misleading as to which component was absent - those were easily Googleable, but it would be helpful if the actual solution could be output at build time. But finally I get to here:

• installing source package ‘sf’ ...
  ** package ‘sf’ successfully unpacked and MD5 sums checked
  ** using staged installation
  configure: loading site script /usr/share/site/x86_64-unknown-linux-gnu
  configure: CC: gcc
  configure: CXX: g++ -std=gnu++11
  checking for gdal-config... /usr/bin/gdal-config
  checking gdal-config usability... yes
  configure: GDAL: 3.2.3
  checking GDAL version >= 2.0.1... yes
  checking for gcc... gcc
  checking whether the C compiler works... yes
  checking for C compiler default output file name... a.out
  checking for suffix of executables...
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether we are using the GNU C compiler... yes
  checking whether gcc accepts -g... yes
  checking for gcc option to accept ISO C89... none needed
  checking how to run the C preprocessor... gcc -E
  checking for grep that handles long lines and -e... /usr/bin/grep
  checking for egrep... /usr/bin/grep -E
  checking for ANSI C header files... yes
  checking for sys/types.h... yes
  checking for sys/stat.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for memory.h... yes
  checking for strings.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for unistd.h... yes
  checking gdal.h usability... yes
  checking gdal.h presence... yes
  checking for gdal.h... yes
  checking GDAL: linking with --libs only... yes
  checking GDAL: /usr/share/gdal/pcs.csv readable... no
  checking GDAL: checking whether PROJ is available for linking:... yes
  checking GDAL: checking whether PROJ is available fur running:... yes
  configure: GDAL: 3.2.3
  configure: pkg-config proj exists, will use it
  configure: using proj.h.
  configure: PROJ: 7.2.1
  checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
  configure: error: libproj or sqlite3 not found in standard or given locations.
  ERROR: configuration failed for package ‘sf’
• removing ‘/usr/lib64/R/library/sf’

The downloaded source packages are in
‘/tmp/RtmpLD4faj/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Warning message:
In install.packages("sf") :
installation of package ‘sf’ had non-zero exit status

r:~ # ldconfig -p | grep sqlite
libsqlite3.so.0 (libc6,x86-64) => /lib64/libsqlite3.so.0
r:~ # ldconfig -p | grep proj
libproj.so.19 (libc6,x86-64) => /lib64/libproj.so.19
r:~ # rpm -qa | grep sqlite
libsqlite3-0-3.35.5-1.2.x86_64
akonadi-server-sqlite-21.04.1-1.2.x86_64
sqlite3-3.35.5-1.2.x86_64
sqlite3-devel-3.35.5-1.2.x86_64
libQt5Sql5-sqlite-5.15.2-6.1.x86_64
php7-sqlite-7.4.20-1.1.x86_64
r:~ # rpm -qa | grep proj
proj-data-at-7.2.1-2.2.noarch
proj-data-br-7.2.1-2.2.noarch
libproj19-7.2.1-2.2.x86_64
proj-data-ca-7.2.1-2.2.noarch
proj-data-ch-7.2.1-2.2.noarch
proj-data-de-7.2.1-2.2.noarch
proj-data-dk-7.2.1-2.2.noarch
proj-data-es-7.2.1-2.2.noarch
proj-data-eur-7.2.1-2.2.noarch
proj-data-fi-7.2.1-2.2.noarch
proj-data-jp-7.2.1-2.2.noarch
proj-data-au-7.2.1-2.2.noarch
proj-devel-7.2.1-2.2.x86_64
proj-7.2.1-2.2.x86_64
proj-data-be-7.2.1-2.2.noarch
proj-data-fo-7.2.1-2.2.noarch
proj-data-fr-7.2.1-2.2.noarch
proj-data-is-7.2.1-2.2.noarch
proj-data-nc-7.2.1-2.2.noarch
proj-data-nl-7.2.1-2.2.noarch
proj-data-nz-7.2.1-2.2.noarch
proj-data-pt-7.2.1-2.2.noarch
proj-data-se-7.2.1-2.2.noarch
proj-data-sk-7.2.1-2.2.noarch
proj-data-uk-7.2.1-2.2.noarch
proj-data-us-7.2.1-2.2.noarch

In the "along the way to here" wack-a-mole resolving dependencies, also noticed the auto config will complain about GDAL and later PROJ when installed, but what it really wants is for at least some proj-data-XX packages also to be installed and that would be not obvious.

Is this a missing dependency on the host or is there some way an end user or I can specify, explicitly, the locations of the libraries? I'm normally pretty good at figuring this stuff out, but have just run aground on this one.

issue also observed on Leap 15.2 and 15.3 ( latest )

-- Bob

[edzer]

We need to see the output of the error; could you pls follow the instructions here: #1685 (comment) ?

[Enchufa2]

Same issue in Fedora 34 and rawhide (no issues though following the instructions referenced in the comment above).

[Enchufa2]

According to the build history, the package started to fail with v0.9-8, so I believe that this is due to --static being added in this commit: c085e2a#diff-49473dca262eeab3b4a43002adb08b4db31020d190caaad1594b47f1d5daa810R388. What is this flag for?

[Enchufa2]

Confirmed: the --static flag is the culprit. Builds in Fedora 33, 34 and rawhide succeeded without it: https://copr.fedorainfracloud.org/coprs/iucar/cran/build/2295538/

[rsbivand]

But probably only because proj.pc is IMO malformed, omits -ltiff -lcurl -lsqlite3 and uses the specific *.so files instead, see: https://bugzilla.redhat.com/show_bug.cgi?id=1958191. With the -l* specification, both static and dynamic work, with just the SOs, static cannot work. The --static flag in configure.ac is useful to test for static-build platforms. sf has the flag, rgdal does not, but does add some -l* in configure tests. If proj.pc was vanilla (as simply built from source), it would use -l*, not the SOs. The configure.ac files have to match many platforms, including static build ones.

[Enchufa2]

But probably only because proj.pc is IMO malformed, omits -ltiff -lcurl -lsqlite3 and uses the specific *.so files instead, see: https://bugzilla.redhat.com/show_bug.cgi?id=1958191. With the -l* specification, both static and dynamic work, with just the SOs, static cannot work.

It's definitely odd, but Fedora/EPEL packaging does nothing special, it just calls cmake as indicated by upstream. So this issue comes from upstream.

Anyway, it doesn't matter, because Fedora/EPEL packages apparently do not provide a static libproj.a since v7.2.0 (maybe it's broken?), and it seems that openSUSE doesn't either.

The --static flag in configure.ac is useful to test for static-build platforms. sf has the flag, rgdal does not, but does add some -l* in configure tests. If proj.pc was vanilla (as simply built from source), it would use -l*, not the SOs. The configure.ac files have to match many platforms, including static build ones.

So let the platform choose. If a certain platform only has shared libraries, why should you impose testing a static build? Or the other way around. The platform will add --static to the flags when needed, I don't think this should be part of your configure.

[rsbivand]

Maybe autotools builds proj.pc one way, cmake another way? I avoid cmake whenever possible, even after turning off ccache. PROJ source INSTALL does not mention cmake, I'm trying a cmakebuild now with PROJ 8.0.1 (current); 7.2.* only makes sense for downstream packages still using proj_api.h.

Reporting back on source cmake build of PROJ 8.0.1 on F34; make check does not work; proj.pc seems to be as for autotools:

prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib64
includedir=${prefix}/include
datarootdir=${prefix}/share
datadir=${datarootdir}/proj

Name: PROJ
Description: Coordinate transformation software library
Requires:
Version: 8.0.1
Libs: -L${libdir} -lproj
Libs.private: -lsqlite3 -ltiff -lcurl -lstdc++
Cflags: -I${includedir}

So maybe it isn't "upstream", that is not PROJ, but some choice after downloading the PROJ tarball.

[Enchufa2]

Fedora 34 has v7.2.1 and it's not gonna update to v8.x.x. Fedora rawhide does have v8.0.1, and I see the same, so it was a bug in v7.2.x and upstream already fixed it for v8.x.x (BTW, it's ctest what you should use after a cmake build, instead of make check).

Anyway, sf fails in Fedora rawhide, because there is no static library available. So I still think that sf should not try to force --static in any platform.

[rsbivand]

OK. GEOS cmake admits make check, I simply followed that lead. I agree that --static in sf/configure.ac is not essential, but disagree that downstream (sf) needs to special case RPMs excluding static libraries - why would a devel package exclude them? @edzer does debian bundle static libraries?

[Enchufa2]

I agree that --static in sf/configure.ac is not essential, but disagree that downstream (sf) needs to special case RPMs excluding static libraries - why would a devel package exclude them? @edzer does debian bundle static libraries?

I opened an issue to ask why we are not providing static libs anymore (https://bugzilla.redhat.com/show_bug.cgi?id=1975265), but the truth is that the SPEC file does nothing special, does not exclude anything. So I suppose that the new cmake-based build system does not create the static lib by default, and two separate builds are necessary (in which case this issue is likely to reappear).

[rsbivand]

Yes, the problem is that the cmake in PROJ did not build the static libraries (also here, but maybe my incantation was acanonical). Shall I or will you open an issue with PROJ? I really am clueless about cmake, so I cannot suggest a PR.

[edzer]

@edzer does debian bundle static libraries?

Ubuntu 20.04 with ubuntugis-unstable PPA has both, static and dynamic proj libraries installed.

[rsbivand]

Maybe Debian/Ubuntu use autotools, not cmake? Does their proj.pc show -l* rather than SOs?

[Enchufa2]

Yes, the problem is that the cmake in PROJ did not build the static libraries (also here, but maybe my incantation was acanonical). Shall I or will you open an issue with PROJ? I really am clueless about cmake, so I cannot suggest a PR.

cmake does not produce both versions in one go. Two builds are mandatory if both versions are required.

Maybe Debian/Ubuntu use autotools, not cmake?

Debian/Ubuntu still use autotools, yes, but autotools support will be dropped eventually in proj.

Also, proj's maintainer for Fedora notes that our guidelines say that packages SHOULD NOT ship static libraries. openSUSE's guidelines go in the same direction. And Mageia.

[rsbivand]

I closed the PROJ issue, thanks for clarification @Enchufa2 . The --static will either have to be dropped, or dropped conditional on OS and packaging mechanism. Replacing autotools by cmake by the packagers isn't controlled by us, nor is a preference not to ship static libraries by the packagers.

[edzer]

@rsbivand do you remember why --static was added, and by whom?

[Enchufa2]

Yes, exactly, apologies if I wasn't clear before about cmake behaviour. The policy about static libraries didn't change lately. What happens (I suppose) is that, with autotools, it was easier to just ship everything, while now, with cmake, it's easier to avoid the static version. And given that the guidelines recommend to avoid static libraries, probably the packager will be unwilling to perform two builds to go against this guideline.

All in all, I think that dropping --static here, and leaving that choice to the system that builds sf, is the most straightforward solution.

@rsbivand do you remember why --static was added, and by whom?

@edzer You did! :D :D Here: c085e2a#diff-49473dca262eeab3b4a43002adb08b4db31020d190caaad1594b47f1d5daa810R388

[rsbivand]

Seems to be a change 17 January this year. I don't know the context.

[edzer]

Trying reversal in #1702

[Bob-O-Rama]

r:/build/sf_proj_issue # vi proj_test.cpp

r:/build/sf_proj_issue # cat proj_test.cpp
#include <stdio.h>
#include <stdlib.h>
#include <proj.h>

int main() {
proj_context_create();
exit(0);
}

r:/build/sf_proj_issue # g++ -o proj_test proj_test.cpp -lproj -lsqlite3

r:/build/sf_proj_issue # ls
proj_test proj_test.cpp

r:/build/sf_proj_issue # ls -lia
total 28
3874199 drwxr-xr-x 1 root root 44 Jun 23 12:07 .
326385 drwxr-xr-x 1 root root 1062 Jun 23 12:05 ..
3874205 -rwxr-xr-x 1 root root 23864 Jun 23 12:07 proj_test
3874203 -rw-r--r-- 1 root root 113 Jun 23 12:06 proj_test.cpp

r:/build/sf_proj_issue # ./proj_test

r:/build/sf_proj_issue #

Builds and executes successfully.

r:/build/sf_proj_issue # ldd ./proj_test
linux-vdso.so.1 (0x00007ffe125d7000)
libproj.so.19 => /lib64/libproj.so.19 (0x00007f411e757000)
libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x00007f411e60e000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f411e3f5000)
libm.so.6 => /lib64/libm.so.6 (0x00007f411e2b1000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f411e296000)
libc.so.6 => /lib64/libc.so.6 (0x00007f411e0c7000)
libtiff.so.5 => /lib64/libtiff.so.5 (0x00007f411e040000)
libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f411df9e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f411df7d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f411df76000)
/lib64/ld-linux-x86-64.so.2 (0x00007f411eaa2000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f411de82000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f411de4f000)
libjbig.so.2 => /lib64/libjbig.so.2 (0x00007f411de3f000)
libjpeg.so.8 => /lib64/libjpeg.so.8 (0x00007f411ddaf000)
libz.so.1 => /lib64/libz.so.1 (0x00007f411dd95000)
libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f411dd6b000)
libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f411dd4a000)
libssh.so.4 => /lib64/libssh.so.4 (0x00007f411dcdc000)
libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f411dcc6000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f411dc2f000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f411d93f000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f411d8eb000)
libldap_r-2.4.so.2 => /lib64/libldap_r-2.4.so.2 (0x00007f411d893000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f411d882000)
libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f411d873000)
libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f411d6ee000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f411d61f000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f411d606000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f411d600000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f411d5ee000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f411d5d4000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f411d5b5000)
libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f411d592000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f411d58b000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f411d55d000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f411d4b0000)

[edzer]

Thank you; I now found the source for adding --static, which was here: r-spatial/lwgeom#64 - it solved a problem on osx builds.

[Enchufa2]

Per @jeroen's comment there, the solution was to install libsqlite3, libcurl and libtiff too and add the proper linker flags, -lsqlite3 -ltiff -lcurl in their own private Makevars. Other packages using PROJ do not have --static and build just fine on OSX platforms on CRAN, so that makes me think that this was a particular misconfiguration for that user. I don't think then that imposing --static for everybody is a good solution.

[edzer]

Thanks; I now asked there whether and how --static can be constrained to be used for OSX only.

[rsbivand]

See also a helpful comment by @mwtoews in the PROJ issue OSGeo/PROJ#2753 (comment): OSGeo/PROJ#2546 (comment)

[edzer]

I think this should do it. Anything against merging in main branch?

[Enchufa2]

Also, let me raise a side, but related, question. Why all the r-spatial packages declare SQLite3 as a system requirement? It is only needed because the configure script looks for it, but then none of these packages, as far as I've seen, end up linked against SQLite3.

[edzer]

I agree that the test in question may have a higher cost than benefit. We could for instance use a standard configure construct that would only check whether linking with -lproj works, I guess that would satisfy our needs, @rsbivand ?

[Enchufa2]

I could be missing something, of course, but the Requires have no dependency on SQLite3, as you can see:

Processing files: R-CRAN-sf-1.0.0-1.fc35.copr2295538.x86_64
Provides: R-CRAN-sf = 1.0.0-1.fc35.copr2295538 R-CRAN-sf(x86-64) = 1.0.0-1.fc35.copr2295538
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: libR.so()(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3.4)(64bit) libc.so.6(GLIBC_2.32)(64bit) libc.so.6(GLIBC_2.4)(64bit) libgcc_s.so.1()(64bit) libgcc_s.so.1(GCC_3.0)(64bit) libgcc_s.so.1(GCC_3.3.1)(64bit) libgdal.so.29()(64bit) libgeos_c.so.1()(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit) libproj.so.22()(64bit) libstdc++.so.6()(64bit) libstdc++.so.6(CXXABI_1.3)(64bit) libstdc++.so.6(GLIBCXX_3.4)(64bit) libstdc++.so.6(GLIBCXX_3.4.11)(64bit) libstdc++.so.6(GLIBCXX_3.4.14)(64bit) libstdc++.so.6(GLIBCXX_3.4.20)(64bit) libstdc++.so.6(GLIBCXX_3.4.21)(64bit) libstdc++.so.6(GLIBCXX_3.4.26)(64bit) libstdc++.so.6(GLIBCXX_3.4.29)(64bit) libstdc++.so.6(GLIBCXX_3.4.9)(64bit) rtld(GNU_HASH)

and -lsqlite3 is not even used by the linker. In proj4, @s-u does add -lsqlite3 to LIBS, but then again, the shared library is not linked against it, because it's not really needed:

Processing files: R-CRAN-proj4-1.0.10.1-1.fc35.copr2297063.x86_64
Provides: R-CRAN-proj4 = 1.0.10.1-1.fc35.copr2297063 R-CRAN-proj4(x86-64) = 1.0.10.1-1.fc35.copr2297063
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: libR.so()(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.4)(64bit) libproj.so.22()(64bit) rtld(GNU_HASH)

[rsbivand]

In rgdal configure.ac, I see:

dnl PROJ >= 6 needs C++ runtime and libsqlite3 (and perhaps more)
dnl when statically linked.
${CXX} ${CPPFLAGS} ${PKG_CPPFLAGS} -o proj_conf_test2 proj_conf_test2.cpp ${PKG_LIBS} -lproj -lsqlite3

These were added during January 2020, but there is much more in emails to @edzer and me from Brian Ripley of 24 May 2020 and 10 June 2020 (and the next few days).

[edzer]

AFAICT it's only used (explicitly) for making this test run, and I don't see really what would go wrong if we get rid of this test altogether. I do know that the issue of missing libsqlite3-dev or so came up many times, not only here.

[Enchufa2]

In general, libA depends on libB, libC, libD, then pkgA depends on libA, so it should declare a dependency on libA. If a particular build wants to link statically, then of course it would need libA's dependencies to be present too: libB, libC, libD.

Here, libsqlite3 is required if one wants to statically link against PROJ, as rgdal's comment points out. But so is libtiff, libcurl and libstdc++, as per

$ pkg-config --libs --static proj
-lproj -lsqlite3 -ltiff -lcurl -lstdc++

However, no package linking against PROJ declares a dependency on libtiff or libcurl, and certainly no one declares a dependency on libstdc++.

[rsbivand]

Would it be possible to add a configure argument to build static, and set up CI only running ./configure for the affected systems (without the argument)? I'm assuming then that Windows and MacOS (building static) could use the same mechanism with the argument to check that new versions do not extend the external dependencies. In general, however, package binaries in systems with dynamic external dependencies should not need libB, libC etc. tests. I think the original reports were misguided, because if PROJ is properly installed, it will find its upstream external dependencies. In at least one case, I think I recall that ldconfig had not been run after a source install of PROJ.

[edzer]

Possible yes, but I find it highly desirable to reduce configure, rather than add to it.

[edzer]

See #1471