Consuming an external package that contains an cmake config-file package

[bldrvnlw]

I also have a case where the conan package (hosted in a private artifactory) provides its own config.cmake files. However in that case the package is not a build requirement but a regular requirement of the code being built. Are you saying that the only way to get this "included" is to declare it as a build requirement?

Hi @bldrvnlw

Using the internal package xxxx-config.cmake is still possible, that is only restricted in ConanCenter, because you should be aware of 2 main CMake limitations regarding xxxx-config.cmake:

• Internal package xxx-config.cmake will not work in multi-config environments, because CMake cannot locate or cannot do generator expressions over find_package(). So if you want to switch Debug/Release configuration from Visual Studio or XCode IDEs, that will not work
• Internal package xxx-config.cmake often have problems with transitivity, and they might find a transitive dependency in the system, instead of a Conan package.

If despite these limitations you still want to use the internal xxx-config.cmake, you can do it, by avoiding the CMakeDeps generator altogether, as you don't want Conan to generate the xxxx-config.cmake files for the dependencies. The CMakeToolchain might still be useful and it is intended to allow locating the xxx-config.cmake inside the packages too.

This discussion is mostly about the extra cmake modules that the CMakeDeps generator will be injecting, which is a different issue than wanting to use your internal xxx-config.cmake modules.

Please let me know if this helps.

@SeanSnyders yes, we know it is still not very clear, we are still heavily working and changing things in all this area, that is the reason the docs are still scarce and confusing. We are trying to stabilize and provide full examples (like conan-io/examples#90)

Originally posted by @memsharded in #9112 (comment)

[memsharded]

Hi @bldrvnlw

This is one of my comments in a previous issue, if you could please clarify what is the thing that needs to be checked, or what is the question or problem, that would help.

[bldrvnlw]

Hi @memsharded I opened this as a separate question to go into more depth. The background here is a conan centric CI with well controlled versions and a more flexible desktop (research) developer environment that is conan free using multi-config IDEs i.e. Visual Studio (and some XCode)

You noted two problems using packages that contain their own cmake Config-file :

1. switching Debug/Release configuration from Visual Studio or XCode IDEs, that will not work
2. Internal package xxx-config.cmake often have problems with transitivity,

I agree with 2. and am looking for a workaround (see below) but am confused by 1. For my Foo package the cmake config-file contains roughly this sort of cmake code:

add_library(Foo STATIC IMPORTED)
set_target_properties(Foo PROPERTIES IMPORTED_LOCATION_DEBUG "/path/to/foo-d.lib")
set_target_properties(Foo PROPERTIES IMPORTED_LOCATION_RELEASE "/path/to/foo.lib")
set_property(TARGET Foo APPEND PROPERTY IMPORTED_CONFIGURATIONS DEBUG)
set_property(TARGET Foo APPEND PROPERTY IMPORTED_CONFIGURATIONS RELEASE)

(This actual code is generated using cmake INSTALL EXPORT and the CMakePackageConfigHelpers tools so is a bit more verbose. )

So basically the following link is correct for debug or release.

target_link_libraries(MyProj foo)

Am I missing something?

To solve point 2. what I'd like to do is continue to use the prepackaged multi-config Foo library (that is the same stable release version used by the desktop developers) but create two, or more, conan installer packages that take care of the pulling the correct Foo transitive dependencies including the the prepackaged lib for the CI conan environment.

The desktop developers who use Foo but are freer in their choice of the exact version of the transitive dependencies and may prefer to build them themselves (for obscure research type reasons). They don't use conan, and I cannot mandate that (nor honestly do I wish to).

The packages would be

conan_foo (for Release) ->
                                        Foo
                                        + Foo release dependencies 

conan_foo (for debug) ->
                                        Foo
                                        + Foo debug dependencies 

Possibly more - eg. RelWithDebugInfo

Foo is also currently built using conan but with no build_type. That will not be supported in the future by conan so it might have a dummy type say just Release but continue to contain all the configs.

Is there a way to do this?

[memsharded]

The problem is locating and loading xxx-config.cmake files from different locations. The find_package() CMake model is limited to 1 location. Conan packages implement 1 package for every binary, in different locations, for many reasons:

• It doesn't scale to have just 1 location to store an increasingly number of binaries Release, Debug, RelWithDebInfo, etc
• It is not safe to include some Debug information in some packages that are to be distributed, deployed, etc.

The above cmake code:

add_library(Foo STATIC IMPORTED)
set_target_properties(Foo PROPERTIES IMPORTED_LOCATION_DEBUG "/path/to/foo-d.lib")
set_target_properties(Foo PROPERTIES IMPORTED_LOCATION_RELEASE "/path/to/foo.lib")
set_property(TARGET Foo APPEND PROPERTY IMPORTED_CONFIGURATIONS DEBUG)
set_property(TARGET Foo APPEND PROPERTY IMPORTED_CONFIGURATIONS RELEASE)

Assumes that:

• You got strictly 1 xxx-config.cmake that represents all possible binaries
• Typically the binaries, etc, might have a common root, for includes, libs, etc.

The second is not a problem if everything is made abs paths, but the first one is a real blocker.

The generated files by CMakeDeps include all of the logic that allows to do the switch between any built-in and custom CMake configurations, but this is impossible with standard CMake generated config files.

Does this make sense?

[bldrvnlw]

Actually the CMake generated xxxTargets-config.cmake at the top level is a bit more elegant than a single file. It GLOBs and includes all the configuration FooTargets-*.cmake files (where * is debug/release/relwithdebuginfo etc) thus:

file(GLOB CONFIG_FILES "${_DIR}/FooTargets-*.cmake")
foreach(f ${CONFIG_FILES})
  include(${f})
endforeach()

Regarding the second point: indeed the binaries have a common include root in this case.

But I'm not trying to argue against the the Conan vision of having a separate package for each configuration, it just in this case the CMake provided packaging config-file suits our needs very well and I'd much prefer to stick with it.

However regarding the second question is there a way to point from a package to a dependency with different build_type settings that will be future-proof w.r.t. conan 2.0? Profiles perhaps? I'm guessing that the two packages must have a different name to avoid loop issues.

[memsharded]

Ok, I think I need to step back and understand the big picture. Lets see if I got it right so far:

• You have one "Foo" Conan package that contains, in 1 single package binary, different build_type configurations: Release, Debug, RelWithDebInfo.
• I understand that you when you build this package, you iterate those configurations in the build() method, building each one and packaging all of them?
• It is fine to remove or not include the build_type setting if you dont use it.
• This package contains some ```foo-config.cmakethat you want to use instead of the ConanCMakeDeps`` automatically generated one. This ``foo-config.cmake`` will work for all configurations, as they are all in the same package.

Some questions:

• Do developers install other packages beside the Foo one? The other packages are regular Conan packages and you want the CMakeDeps generated files or not?
• Are the Foo transitive dependencies typical Conan packages with different binaries for Debug, Release, etc?
• are you aware of the property skip_deps_file? https://github.com/conan-io/docs/pull/2121/files (added in Conan 1.38). It avoids creating the files (and then allowing using the in-package ones).

[bldrvnlw]

Hi @memsharded - yes to all 4 four clarification points that's exactly the situation.

I'll answer the three questions in detail:

1. Yes developers install other packages. Most developers work own windows (VS) & a few on macos (XCode) so that actually means the "hard-core" C++ developers build everything themselves.

      However some developers are application focussed and prefer an easier development path. This group prefers prebuilt packages. As well as package Foo (low-level data analysis) there is a high-level Core package for a UI plugin system which has a QT requirement which developers install themselves (the version of Qt is clearly defined project wide). For those application focussed users I created the CMake config-file for Foo and also its dependencies, and for Core.

2. The Foo transitive dependencies are typical conan dependencies with different binaries for debug, release. While building the Foo CMake package on the CI conan I use the cmake_multi generator and the conanbuildinfo_multi.cmake is injected into the Foo CMake project. The could probably be beter done with CMakeDeps now. For the developers I also make a multi-config cmake package with these dependencies.

3. No I didn't know about the skip_deps_file. I've read the doc on this new feature now and I think it may help, but I need to look at the process in detail.

[bldrvnlw]

I've put together a test repo to run through the situations I want to support, conan and non-conan builds. I get very close but the skip_deps_file has one minor "issue"

Normally the CMakeDeps class (https://github.com/conan-io/conan/blob/develop/conan/tools/cmake/cmakedeps/cmakedeps.py) creates a number of files based on templates.

CMakeDeps (this just lists the features relevant to the problem) :

• creates [PACKAGE_NAME]BUILD_DIRS[CONFIG] (in the [PACKAGE_NAME]-[CONFIG]-x86_64-data.cmake file generated by ConfigDataTemplate)
• adds [PACKAGE_NAME]BUILD_DIRS[CONFIG] to the CMAKE_MODULE_PATH and CMAKE_PREFIX_PATH (in [PACKAGE]Target-[CONFIG].cmake generated by TargetConfigurationTemplate)

To reuse the CMake package config-file in the skip_deps_file package these two steps are still needed in order for the CMake find_package to work.

Would it be possible to have a different functionality to skip_deps_file perhaps called use_package_deps where only the required updates to CMAKE_MODULE_PATH and CMAKE_PREFIX_PATH are generated using the [PACKAGE_NAME]BUILD_DIRS[CONFIG]?

In that case the CMake find package should find the package included package config-file.

[bldrvnlw]

Leaving the skip_deps_file funcationality as is, the following is a first attempt at injecting the package root into the conan_toolchain.cmake file which permits the find_package to work on the package locale cmake file-config

    def _get_package_path(self, package_str):
        stream = StringIO()
        output = ConanOutput(stream)
        conan_api = Conan(output=output)
        command = Command(conan_api)
        command.run(["info", f"{package_str}@", "--paths"])
        outstr = stream.getvalue()
        package_path = package_str.replace("/", r"\\")
        pat = re.compile(fr"\n[ ]*package_folder: (.*{package_path}.*)\n")   # group capture raw string preserve \\
        x = pat.search(outstr)  # first occurrence
        package_root = x.groups()[0]  # e.g. 'C:\\Users\\bvanlew\\.conan\\data\\Foo\\0.2.0\\_\\_\\package\\f34583babc53eea864e76087e72c782c95f0f402'
        return package_root.replace("\\", "/")

    def _inject_package_root(self, package_str):
        package_root = self._get_package_path(package_str)
        with open("conan_toolchain.cmake", "a") as toolchain:
            toolchain.write(fr"""
set(CMAKE_MODULE_PATH "{package_root}" ${{CMAKE_MODULE_PATH}})
set(CMAKE_PREFIX_PATH "{package_root}" ${{CMAKE_PREFIX_PATH}})
            """)

    def generate(self):
        print("In generate")
        tc = CMakeToolchain(self)
        tc.generate()
        deps = CMakeDeps(self)
        deps.generate()
        self._inject_package_root(r"Foo/0.2.0")

[bldrvnlw]

My demo repo creating the Foo package is here https://github.com/bldrvnlw/Foo it contains one non-conan consumer (Bar) and a conan consumer (Car).

As well as the requirements free pure CMake package Foo, a conan compatible package Foo_deps is created at the same time for use in the conan based consumer. The conan based consumer used the _inject_package_root trick to locate and place the Foo package CMake file in the CMAKE_PREFIX_PATH.

[memsharded]

Calling from a Conan recipe the python api is forbidden, it can easily lead to many undefined behavior, please don't do it: command.run(["info", f"{package_str}@", "--paths"])

For a start, there is nothing that forces there that the same settings used for the current evaluation will be the same evaluated by the nested conan info, so it could very easily give you the path of another (incompatible) binary package.

If you need to access the dependencies, try via self.dependencies["pkg"].package_folder, but never using internal Conan implementation classes (ConanOutput, Command, Conan)

[bldrvnlw]

@memsharded Hi James. Thanks for the warning, I was thinking in terms of the command line and had forgotten the availability of self.dependencies.

For a generic solution I've added an extra property cmake_config_file to be set on a package that contains it's own cmake config-file. This should be set alongside the standard skip_deps_file in the consumer package conanfile.py thus

            self.cpp_info.set_property("skip_deps_file", True)
            self.cpp_info.set_property("cmake_config_file", True)

Then in the consumer I have

    def fix_config_packages(self):
        """ Iterate the dependencies and add the package root where
        it is marked as a "cmake_config_file" and when "skip_deps_file" is
        enabled. Permits using a package locak cmake config-file.
        """
        package_names = {r.ref.name for r in self.dependencies.host.values()}
        for package_name in package_names:
            cpp_info = self.dependencies[f"{package_name}"].new_cpp_info
            if (cpp_info.get_property("skip_deps_file", CMakeDeps) and
                    cpp_info.get_property("cmake_config_file", CMakeDeps)):
                package_root = Path(self.dependencies[
                    f"{package_name}"].package_folder)
                with open("conan_toolchain.cmake", "a") as toolchain:
                    toolchain.write(fr"""
set(CMAKE_MODULE_PATH "{package_root.as_posix()}" ${{CMAKE_MODULE_PATH}})
set(CMAKE_PREFIX_PATH "{package_root.as_posix()}" ${{CMAKE_PREFIX_PATH}})
                    """)

    def generate(self):
        print("In generate")
        tc = CMakeToolchain(self)
        tc.generate()
        deps = CMakeDeps(self)
        deps.generate()
        self.fix_config_packages()

That meets my requirements but it probably does not cover all cases (e.g. build requires)

[memsharded]

I think this issue got response in https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/use_package_config_cmake.html and newer Conan 2 features some time ago, closing it as solved, but please open new tickets for any further question or issue if necessary, thanks for the feedback!