gdal: add v3.9.0, fix a CMake bug, bump deps

[valgur]

The custom find_package2() failed to set non-capitalized dependency vars (e.g. PostgreSQL_LIBRARIES) in a globally visible manner. Fixed this.

[franramirez688]

@valgur It looks good, but I'd like to see any log using those options related to the bumped dependencies (the remaining ones defaulted to False). It's only to verify that we're not introducing any conflicts or raising different problems.

[valgur]

@franramirez688

It's only to verify that we're not introducing any conflicts

That's almost impossible at the number of dependencies this library supports. Until we have a method of bumping each dependency simultaneously across the whole CCI at least.

Other than that, can do.

[franramirez688]

@valgur I just tried it. I got some conflicts:

ERROR: Version conflict: libspatialite/5.1.0->libxml2/2.12.4, gdal/3.8.4->libxml2/2.12.5.

ERROR: Version conflict: Conflict between expat/2.5.0 and expat/2.6.2 in the graph.
Conflict originates from freexl/2.0.0

ERROR: Version conflict: openexr/3.2.3->imath/3.1.9, gdal/3.8.4->imath/3.1.10.

ERROR: Version conflict: libtiff/4.6.0->xz_utils/5.4.5, gdal/3.8.4->xz_utils/5.6.1.

Anyway, applying this diff was enough to avoid it:

diff --git a/recipes/gdal/post_3.5.0/conanfile.py b/recipes/gdal/post_3.5.0/conanfile.py
index 7418aa855f..9fb263c22d 100644
--- a/recipes/gdal/post_3.5.0/conanfile.py
+++ b/recipes/gdal/post_3.5.0/conanfile.py
@@ -207,10 +207,10 @@ class GdalConan(ConanFile):
         if self.options.with_ecw:
             self.requires("libecwj2/3.3")
         if self.options.with_expat:
-            self.requires("expat/2.6.2")
+            self.requires("expat/2.5.0")
         if self.options.with_exr:
             self.requires("openexr/3.2.3")
-            self.requires("imath/3.1.10")
+            self.requires("imath/3.1.9")
         if self.options.with_freexl:
             self.requires("freexl/2.0.0")
         if self.options.with_geos:
@@ -246,7 +246,7 @@ class GdalConan(ConanFile):
         if self.options.with_libkml:
             self.requires("libkml/1.3.0")
         if self.options.with_lzma:
-            self.requires("xz_utils/5.6.1")
+            self.requires("xz_utils/5.4.5")
         if self.options.with_lz4:
             self.requires("lz4/1.9.4")
         if self.options.with_mongocxx:
@@ -295,7 +295,7 @@ class GdalConan(ConanFile):
         if self.options.with_xerces:
             self.requires("xerces-c/3.2.5")
         if self.options.with_xml2:
-            self.requires("libxml2/2.12.5")
+            self.requires("libxml2/2.12.4")
         if self.options.with_zstd:
             self.requires("zstd/1.5.5")
         # Use of external shapelib is not recommended and is currently broken.

Another related one was:

ERROR: Provide Conflict: Both 'libjpeg/9e' and 'libjpeg-turbo/3.0.2' provide 'None'.

This one was caused by using -o with_jpeg="libjpeg-turbo" because libtiff default option is self.options.jpeg = "libjpeg".

[valgur]

@franramirez688
Thanks for experimenting. I needed 6 force-s and 3 override-s to get all optional dependencies to work. The above patch was not sufficient.

Also, basisu, ecw, poppler and netcdf were broken, but that's a rather low number. netcdf is the only concerning one as it's a pretty crucial format for remote-sensing datasets. #21656 did not help either.

That was with static libs. Some dependency broke the shared build pretty badly, though. Edit: this was caused by tools=True. Added a check for that to validate().

[jcar87]

@franramirez688 Thanks for experimenting. I needed 6 force-s and 3 override-s to get all optional dependencies to work. The above patch was not sufficient.

Also, basisu, ecw, poppler and netcdf were broken, but that's a rather low number. netcdf is the only concerning one as it's a pretty crucial format for remote-sensing datasets. #21656 did not help either.

That was with static libs. Some dependency broke the shared build pretty badly, though. Edit: this was caused by tools=True. Added a check for that to validate().

Hi @valgur - thanks for this PR. In recent weeks we have taken the decision to not merge PRs that bump version dependencies without a motivation for each bump.

I'm really happy to hear that you are successfully using overrides to solve version conflicts. However, we keep receiving feedback from users - especially those who are new to Conan Center, conflicts become a problem and it becomes an obstacle. And then of course, the issues that arise in Conan Center CI when a build fails because of pre-existing conflicts that are unrelated to the PR being built.

We have noticed in recent months that indiscriminately bumping dependencies tend to introduce conflicts, rather than fix them - and these are conflicts that can only be solved by other bump dependencies in other PRs. We want to minimise the amount of time any conflict is present with the default options, while we work on a more robust solution.

The team is currently working on a number of strategies to address this moving forward:

• Increase the number of libraries for which we use version ranges - but we need to be careful since not all libraries use a backwards compatible versioning strategy (like semver), and there are still limitations for Conan 1.x users.
• Re-design of the pipeline in a way that allows touching multiple recipes in the same PR, building everything in the right order and consistently (we are already testing this in the background)
• Automate the detection of version conflicts at the PR level, and also ability to find conflicts present in a given commit of the conan-center-index repository, with the ability to automatically generate the changes to fix the conflicts.

That is, moving forward we want a conflict-free experience for most users (at least those who are using a consistent version set provided by us). This is why you may see the team not prioritising version conflicts, or asking them to be removed from a PR that has other fixes. Note however that we will consider version bumps if this fixes an existing conflict, new functionality from a given library is required, or there is a security issue where the new version proposed already has the required fix.

As a note to contributors - given the high load of PRs that we need to review, and bearing any account that bumping dependency versions without justification is not currently prioritised (but by all means, we welcome the PR being open so that we eventually get to it), we would advise PRs that address other issues (e.g. adding a new version, fixing a bug, etc) to be done separately from bumping dependencies without a strong motivator.

[valgur]

@jcar87 @franramirez688
I reverted most of the version bumps in this PR. Consider the remaining ones a follow-up to the previous migration PR (#19298 (comment)), where most of these non-default dependencies were still broken in the first place or still awaiting migration (and a few of the version bumps here still do).

For the more general point, I agree and I'm very grateful for the effort to stabilize things and move forward in a more systematic and efficient manner. The version conflicts are a major paint point on the consumer side. Probably even more than on CCI.

I'll try to revert any non-critical dependency version bumps on my existing PRs and new ones.

However, we still need a stable baseline state with consistent versions across CCI before we can sanely talk about mostly freezing dependency versions. To that end, please direct some of that energy towards and prioritize #23277. It should fix a large portion of version conflicts currently (and frequently) encountered on CCI. I think it's also in line with the likely future approach of handling version bumps by individual dependencies at once instead of doing it in arbitrary and inconsistent fashion across random PRs.

[mayeut]

expat 2.5.0 has some CVEs as mentioned in #23277
expat 2.6.0 doesn't (yet) and is already used in a number of recipes in CCI:

find .. -name conanfile.py -exec grep expat/ {} + | awk '{ print $2" "$1 }' | sort -u
self.requires("expat/2.4.8") ../pangomm/all/conanfile.py:
self.requires("expat/2.5.0") ../aaf/all/conanfile.py:
self.requires("expat/2.5.0") ../apr-util/all/conanfile.py:
self.requires("expat/2.5.0") ../exiv2/all/conanfile.py:
self.requires("expat/2.5.0") ../gdal/post_3.5.0/conanfile.py:
self.requires("expat/2.5.0") ../gdal/pre_3.5.0/conanfile.py:
self.requires("expat/2.5.0") ../gdcm/all/conanfile.py:
self.requires("expat/2.5.0") ../itk/all/conanfile.py:
self.requires("expat/2.5.0") ../jsbsim/all/conanfile.py:
self.requires("expat/2.5.0") ../libarchive/all/conanfile.py:
self.requires("expat/2.5.0") ../libkml/all/conanfile.py:
self.requires("expat/2.5.0") ../libmetalink/all/conanfile.py:
self.requires("expat/2.5.0") ../libvips/all/conanfile.py:
self.requires("expat/2.5.0") ../log4cxx/all/conanfile.py:
self.requires("expat/2.5.0") ../opencolorio/all/conanfile.py:
self.requires("expat/2.5.0") ../openfx/all/conanfile.py:
self.requires("expat/2.5.0") ../readosm/all/conanfile.py:
self.requires("expat/2.5.0") ../xlsxio/all/conanfile.py:
self.requires("expat/2.5.0", ../libstudxml/1.0.x/conanfile.py:
self.requires("expat/2.5.0", ../libstudxml/1.1.x/conanfile.py:
self.requires("expat/2.5.0", ../poco/all/conanfile.py:
self.requires("expat/2.6.0") ../avahi/all/conanfile.py:
self.requires("expat/2.6.0") ../cairo/meson/conanfile.py:
self.requires("expat/2.6.0") ../cpython/all/conanfile.py:
self.requires("expat/2.6.0") ../dbus/1.x.x/conanfile.py:
self.requires("expat/2.6.0") ../fontconfig/all/conanfile.py:
self.requires("expat/2.6.0") ../fontconfig/meson/conanfile.py:
self.requires("expat/2.6.0") ../freexl/all/conanfile.py:
self.requires("expat/2.6.0") ../qt/5.x.x/conanfile.py:
self.requires("expat/2.6.0") ../qt/6.x.x/conanfile.py:
self.requires("expat/2.6.0") ../wayland/all/conanfile.py:
self.tool_requires("expat/2.6.0") ../sdbus-cpp/all/conanfile.py:

I opened #23508 which bumps expat in freexl to 2.6.0 which solves the vulnerable situation and limits conflicts with other recipes.

[valgur]

@mayeut I would move the Expat versions to v2.6.2 as it fixes a CVE as well:
https://github.com/libexpat/libexpat/blob/master/expat%2FChanges
https://nvd.nist.gov/vuln/detail/CVE-2024-28757

[mayeut]

ok, not reflected yet in repology or #23277

[valgur]

@franramirez688 The version bumps part of the PR has been reverted and was not the most significant part of the PR anyway. It fixes some major bugs in the recipe, plus adds a newer version, so a second look (and hopefully a fix for the missing geotiff binary) would be welcome.

[conan-center-bot]

Conan v1 pipeline ✔️

All green in build 6 (7ff3b27eed42836788c2fdbfe5e0e7ccfd623e49):

• gdal/3.8.3:
  All packages built successfully! (All logs)

• gdal/3.9.2:
  All packages built successfully! (All logs)

• gdal/3.7.3:
  All packages built successfully! (All logs)

• gdal/3.5.3:
  All packages built successfully! (All logs)

---

Conan v2 pipeline ❌

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

The v2 pipeline failed. Please, review the errors and note this is required for pull requests to be merged. In case this recipe is still not ported to Conan 2.x, please, ping @conan-io/barbarians on the PR and we will help you.

Failure in build 6 (7ff3b27eed42836788c2fdbfe5e0e7ccfd623e49):

• gdal/3.9.2:
  Didn't run or was cancelled before finishing

• gdal/3.8.3:
  Didn't run or was cancelled before finishing

• gdal/3.5.3:
  Didn't run or was cancelled before finishing

• gdal/3.7.3:
  CI failed to create some packages (All logs)

  Logs for packageID b50fdc221182c22d87f01906f0db93962fca3d1d:

  [settings]
  arch=x86_64
  build_type=Release
  compiler=apple-clang
  compiler.cppstd=17
  compiler.libcxx=libc++
  compiler.version=13
  os=Macos
  [options]
  */*:shared=False
  

  [...]
  deprecated=True
  encryption=False
  fPIC=True
  filesystem_layer=False
  gandiva=False
  hdfs_bridgs=False
  parquet=False
  plasma=deprecated
  runtime_simd_level=max
  shared=False
  simd_level=default
  skyhook=False
  substrait=False
  with_backtrace=False
  with_boost=False
  with_brotli=False
  with_bz2=False
  with_csv=False
  with_cuda=False
  with_flight_rpc=False
  with_flight_sql=False
  with_gcs=False
  with_gflags=False
  with_glog=False
  with_grpc=False
  with_jemalloc=False
  with_json=False
  with_llvm=False
  with_lz4=False
  with_mimalloc=False
  with_openssl=False
  with_opentelemetry=False
  with_orc=False
  with_protobuf=False
  with_re2=False
  with_s3=False
  with_snappy=False
  with_thrift=False
  with_utf8proc=False
  with_zlib=False
  with_zstd=False
  [requires]
  xsimd/9.0.1#c74b1d825f467a37c2a2f03c0b022daa:da39a3ee5e6b4b0d3255bfef95601890afd80709
  
  ERROR: Missing prebuilt package for 'arrow/17.0.0'. You can try:
      - List all available packages using 'conan list "arrow/17.0.0:*" -r=remote'
      - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
      - Try to build locally from sources using the '--build=arrow/17.0.0' argument
  
  More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'
  

---

^{Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.}

[valgur]

Could someone please fix the broken binaries.

The recipe is broken and needs these CMake fixes to work reliably. Please unblock it. The other changes are trivial and largely irrelevant. The version bumps are motivated by new actually working versions becoming available after the recipes for them have been migrated to Conan 2.x. Feel free to revert any that you don't agree with. I don't really care.

Also, the recipe still needs to work around a Conan generators bug via patching, which would also benefit from some attention at some point:

• [bug] CMakeDeps creates targets with dependency cycles conan#15386

[jcar87]

Are the changes in a0480a2 motivated by a bug? I would avoid changes like these if that's not the case.

[valgur]

Are the changes in a0480a2 motivated by a bug? I would avoid changes like these if that's not the case.

Mostly to follow the CMake/Conan conventions: #25381 (comment)

Also, give me a break, please. 😁 This PR, which has been open since March, proposes a fix for an actual major bug that currently breaks a significant portion of dependency handling in the CMake configure step. The tc.variables / tc.cache_variables difference is more or less cosmetic (with some exceptions that should not apply here).

[jcar87]

Mostly to follow the CMake/Conan conventions: #25381 (comment)

I see - that was in response to the addition of CMAKE_POLICY_DEFAULT_CMP0077- where that's an indicator cache_variables should be used. Apologies if this wasn't clear - we don't expect (and actively discourage) amending all existing recipes unless there is an issue - it gives us breathing room both in terms of reviewers (help us direct our attention to the changes that motivate the PR) and CI resources. We have had issues in the pass where things were added/removed/amended as part of something that look like a reformatting - so we do still check, line by line, regardless of how trivial the change ends up being. So I would try and make life a little easier for the reviewing team :)