Feature: model intra-package dependencies

[memsharded]

From #2382

This could help both to model system dependencies self.cpp_info.libs = ["mylib", "pthread"], and then some consumer can depend on pthread by other means, and the order mylib->pthread is not defined.

It can be generalized for your own libs deps "Mylib1"->"MyLib2" inside the same package.

[bschindler]

Do you consider pthread an intra-package dependency? I'm just asking because a number of packages link against system libraries and to me, this doesn't really sound like an intra-package dependency. There are cases (like boost) which provides N libraries itself which may depend on each other in some way and then there are other cases where a package library depends on a system library

[memsharded]

No, that is a system one. But current self.cpp_info.libs model does not support differentiating system and own libraries. We could check if it makes sense to add a self.cpp_info.system_libs, but as the underlying issue is really that the relative order of libraries within a package is not defined, maybe just enabling to define this order will work both for intra-package dependencies and expressing dependencies from package libraries to system libraries as well.

[SSE4]

so, for instance, if we have ffmpeg library which provides the following libraries: libavcodec, libavformat, libavfilter, etc. which might be used independently. so, it would be nice to define in conanfile.py something like this:

self.cpp_info.libavcodec.libs = ['avcodec']
self.cpp_info.libavformat.libs = ['avformat']
self.cpp_info.libavfilter.libs = ['avfilter']

and then, in consumer (e.g. CMake one), we can link to the particular libraries:

target_link_libraries(exe_using_avcodec PRIVATE CONAN_PKG::ffmpeg::libavcodec)
...
target_link_libraries(exe_using_avformat PRIVATE CONAN_PKG::ffmpeg::libavformat)

this is very conceptual, and actual syntax might be very different from my examples, but I guess you got the idea behind it.
same probably applies to lots of other multilib packages (like Qt, wxWidgets, OpenCV, GStreamer, etc.)
these libraries build as a single, and it's not feasible to split each of them into multiple conan files (as code will be the almost the same), like in boost case.

[sigiesec]

This feature should also integrate with frequently used build systems such as CMake. CMake has the information on intra-package dependencies (as well as other information such as public compile options etc.), and it would be very desirable if one wouldn't have to specify this redundantly in the conanfile.py

[memsharded]

Automatically extracting information from CMake about this thing seems a huge challenge for me. Parsing and understanding the scripts files is almost impossible. Using the internal information files generated by cmake could be the way, but also would require a ton of work, and probably still be very fragile wrt cmake different versions, for example, so probably not worth the investment, at least in the first iteration. Having a model to model intra-package dependencies would already be a big and complex feature, so that level of automation won't be possible in the short term.

[sigiesec]

Yes, a CMake integration would of course build on top of that. Maybe it is a good idea to create a separate GitHub issue for that? It is quite a significant problem for us.

Just to give some background on our use case:
We have packages that contain quite a lot of libraries (several packages contains 20-30 libraries, some even more), and we are losing valuable information, which is present in CMake. I am not too familiar with CMake on the other hand. Probably this can be solved on the CMake side in part by generating Find.cmake files that contain the relevant information. At least when using CMake both on the supplier and consumer side, this might also work without modeling this within conan.

[memsharded]

Yes, a CMake integration would of course build on top of that. Maybe it is a good idea to create a separate GitHub issue for that? It is quite a significant problem for us.

Well, until this model has some progress, I think it is not very useful to have such an issue. As I said, it could be very complex and fragile, it is not likely that it will be implemented. So better wait and go step by step.

I think that if they are your own private packages, you might be good using find.cmake files for them, and then you might completely skip the package_info() method. No need to model intra-package dependencies at all, as they will be managed by cmake. The use cases are very different for your own private packages than when creating open-source packages for the community.

[sigiesec]

The use cases are very different for your own private packages than when creating open-source packages for the community.

Well, I think our use cases are a subset of those that need to be supported by open source packages. However, what I sketched about cmake-cmake uses is not really optimal, since this requires the consumer to handle conan packages differently, depending on whether they were built with cmake (and exporting the cmake information) or not. I expect (and in some limited ways conan achieves that already) that a package manager allows to abstract from that, so that I do not need to care about the build systems used to build the packages.

[sigiesec]

I noticed that someone implemented a kind of CMake integration in #112

[memsharded]

Feature needed for @vector-of-bool libman proposal at https://github.com/vector-of-bool/libman.

Related to #3931

[kenfred]

@memsharded I came to this issue from #1323. I'm trying to define multiple cpp_infos like you reasoned about at the end of the thread.

Example:

def package_info(self):
    self.cpp_info.libA.includedirs = ["libA/includes"]
    self.cpp_info.libB.includedirs = ["libB/includes"]

Currently (as of v1.11), the cmake generator will create variables for libA and libB, but will only create a single target. All the target property information is there for the cmake generator to make targets for libA and libB, but the generator isn't making targets for them. This portion of this issue is straight-forward. However, it is being held back by the the additional scope and complexity of intra-package dependencies.

Can we resurrect #1323 to implement the simple target generation? It should be a separate story.

[skgbanga]

Hello,

I think it would be nice if this issue is linked to the getting started page for packaging. Anyone with multiple libs within the same package is going to stumble into this.

Also, I agree with @sigiesec that it seems wasteful to not use the cmake information while building this order. (disclaimer: I don't know sufficient cmake to determine the complexity)

[memsharded]

Hi @kenfred

Could you please clarify what you mean with resurrecting #1323? That feature was already implemented, you can find CONAN_LIB::lib1, CONAN_LIB::lib2 targets for different libraries within a package in the conanbuildinfo.cmake.

Also, I agree with @sigiesec that it seems wasteful to not use the cmake information while building this order. (disclaimer: I don't know sufficient cmake to determine the complexity)

@skgbanga yes, it would be nice, but as commented above, parsing CMake to get this information is almost an impossible challenge, and would be a big source of failing corner cases, and a nightmare in maintenance, support and evolution. Furthermore, all of it would be only useful for CMake build system, but Conan has large usage with many other build systems, and we would need a feature that could be widely applicable to most of them. Defining a subgraph of library relations in a conanfile.py would take only a few minutes, it is mostly one time effort, and easily maintainable and understandable by consumers. It is not worth the effort.

[sigiesec]

parsing CMake to get this information is almost an impossible challenge

I also don't think it is appropriate to parse the CMakeLists.txt files. However, this is also not necessary. CMake already has the option to export the information required by dependents using install(TARGETS), install(EXPORT): https://gitlab.kitware.com/cmake/community/wikis/doc/tutorials/Exporting-and-Importing-Targets#exporting-from-an-installation-tree

The current output of this is CMake code, but it has a specific structure, so it might be parseable already. Or CMake might be extended to output the same information in a well-defined way (JSON, e.g.).

If it were only about the dependencies, the Graphviz output using cmake --graphviz=deps.dot could already be used and parsed quite easily, but additional information is needed (all other library-level interface properties, such as compiler flags/definitions).

we would need a feature that could be widely applicable to most of them

I don't see a contradiction here. Of course, the abstract model in conan must be independent from CMake. You have the option of specifiying the dependencies etc. manually, but you can also implement automatic extractors for specific build systems.

... and a nightmare in maintenance, support and evolution...
Defining a subgraph of library relations ... is mostly one time effort, and easily maintainable

I assume you have a different complexity in mind. For our packages, it would be prohibitively complex to remodel the dependencies in the conanfile, apart from the fact that this introduces significant redundancies, which cannot be more than a workaround.

IMO the reverse is the case: Not having such an automatic extraction/integration with the build system is a nightmare for maintenance, support and evolution of each package.

[kenfred]

@memsharded
I see that now! When I was looking to use the feature, I was expecting it to use CONAN_PKG namespace and not CONAN_LIB, so I missed it in the generated cmake file. Thanks for pointing it out.

[sigiesec]

CMake might be extended to output the same information in a well-defined way (JSON, e.g.).

Such an extension would perfectly fit in with the file API added in cmake 3.14.0: https://cmake.org/cmake/help/v3.14/manual/cmake-file-api.7.html

[memsharded]

This has been closed in PR #6653, but please follow #6716 and #6717 for further development of this feature