[feature] Provide more capabilities for managing sources

[solvingj]

This ticket is intended to aggregate feedback from numerous other tickets related to adding features around managing source files in packages.

Summarized Related Issues

• Enable source caching with out-of-source recipes : [feature] Support exports() and exports_sources() as methods #6943 [question] Backing up an offline Build/Sources of a recipe/package #6876
• Make sources available for downstream debugging : Development sources in project #1796 Enhancement: propogate source file info in package_info #3746
• Make sources available for downstream compilation:
  • LTO and LTCG support Incorrect handling of MSVC versions #3573
    • In ticket above, direction is to completely remodel VS/MSVC tools versioning/etc
    • source compilation propragation are a simpler alternative for supporting LTO/LTCG
  • many orgs share "components" as sources rather than building static libs of everything
  • sqlite amalgamation case: https://www.sqlite.org/amalgamation.html
  • other "unity builds" use cases
  • generated sources use case (some commercial sdk's flexnet/revulytics/codearmor have shared libs + generate sources for you to add to every project that link with them)
  • depend on just the sources of a dependency : examples Enhancement: propogate source file info in package_info #3746 [feature] access deps_cpp_info.src_paths in package() method #6823
    • the boost example was big because depending on boost_python completely changed the dep graph, even though just a single source or two was required from it
  • model dependency tree's which tightly couple with git-submodule references

Summary of salient details coming from discussion:

• With exports_sources, original sources already "cached" and available in all contexts

• Plan to add equivalent "caching" for downloaded sources [feature] Support exports() and exports_sources() as methods #6943

• Original sources are not enough for most use cases : debugging, and downstream compile

  • Will definitely need to enable some strategy in the package() method
  • Generated sources and headers cases, potentially different per settings/options
  • Patches on sources and headers applied during build

• Effectively the same situation surrounding includes()

  • Convention is to package all headers separately for every binary in case they differ slightly
  • Usually fine since header count/size not usually prohibitive
  • Boost/Qt/Some others are exceptional, and the current convention is unfortunate for them

• Do we copy convention from header strategy, or compose exports_sources folder + cpp_info.src_dirs?

• Do we use strictly src_dirs strategy for cpp_info , or can we ALSO enable users to specify explicit list of source files in cpp_info, and then deduce src_dirs from that list. Some debug tools explicitly require a list of search directories, some can use list of sources. Explicit list of sources might provide a more deterministic solution for cases where exports_source directory will have an original source and package might capture a modified or generated alternative.

• I've read the CONTRIBUTING guide.

[memsharded]

A lot of things have happened since this issue was last active:

• Conan 2.0 implements a new way to force the download of sources and execution of source() method, for dependencies, without really needing to build again from source, with the -c tools.build:download_source=True new configuration. It can also be used in conan graph info command, but also in conan install
• The new deployer mechanisms allows to easily collect things from dependencies, including sources, which can be used for preparing some debugging, as they can also be used to automate tasks.

See for example: https://docs.conan.io/2/examples/extensions/deployers/sources/custom_deployer_sources.html

In any case, we have learned that the best for debugging is doing an actual local build, like conan install --build=pkg_to_debug* and that already allows to step-into that dependency.

So I am closing this as solved, please create new tickets for further questions.