recipe id using sub project's latest commit id

[xiaoliangwu]

Hi
if we have multi subproject under same git repo, how to use the latest commit id under each subproject to be the revision_id of each subproject? ( assuming we use revision_mode="scm" )

thanks

[memsharded]

Hi @xiaoliangwu

Thanks for your question.
The Git helper in https://docs.conan.io/2/reference/tools/scm/git.html has a folder argument in the constructor: classGit(conanfile, folder='.'). This folder is the folder that will be used to get the commit from.
The get_commit(), get_url_and_commit() methods will get the commit of the subfolder.

You can see some application of the Git helper in https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html too

[xiaoliangwu]

But , how to set the recipe_id when I get the commit_id

I means, is there a method like:

def recipe_id(self):
     return Git(self, "subdir" ).get_commit()

[memsharded]

There is not such recipe_id that doesn't exist.
If you mean the recipe revision, you can do:

class Pkg(ConanFile):
     revision_mode = "scm"

That will take the git commit as the recipe revision, instead of the revision_mode = "hash" (this is the default, no need to specify it)

[xiaoliangwu]

revision_mode="scm" will always get the latest commit_id from git repo root. for example , I have 2 sub project A and B. I hope A's recipe revision is the latest commit id in A sub folder and B's recipe revision is the latest commit id in B sub folder. current I cannot differentiate them.

let me explain the motivation:

I have a monorepo project , like

root -  module A
    |-  module B
    |-  app

each have it's own conanfile.py for A, B and app . and app depend on A and B. when user push a commit , it could update A or B or app. my idea is we always run "conan create app --build=missing":

in app's conanfile.py:

def requirements(self):
     commit_id_A = Git(self , "A").get_commit()
     self.requires( f"A/1.0@user/testing#{commit_id_A}" )
     commit_id_B = Git(self , "B").get_commit()
     self.requires( f"B/1.0@user/testing#{commit_id_B}" )

so if user update a files in B , the B commit_id will be updated (thus B recipe revision get updated accordingly , but A recipe revision not get changed ) and get rebuild.

in this way, we streamline the dependencies and auto rebuild the modules when any module's source updated.

[memsharded]

Thanks for the explanation @xiaoliangwu

It is correct, the revision_mode = "scm" takes the root of the repo as value.
This is necessary, because it is valid for a conanfile.py not to be located in the root of its repo, but it can be for example in a conan/conanfile.py subfolder. If the used commit was the subfolder one, it would be incorrect in most of the cases, missing the source code changes from the repo.

I think the main issue is that you are trying to have both opposite approaches at the same time, the "monorepo" and the "package management". These approaches are very opposite:

• "package management" means that every package has its own version, can be developed independently and be released independently. But developing it in a mono-repo and coupling it with the other packages in the way that you are trying to force in the requirements() method, means that it is not really that development paradigm
• "monorepo" means all subprojects moves in parallel, they are not versioned independently, they are not released independently with different versions.

Conan has some features, like the editables that allows simultaneously developing more than one separate package. They are still independent, and they typically live in different repos, but this is a mechanism to improve the developer experience in some situations.

So the question could be: what is the underlying pain that you are trying to solve with Conan? If you want to go with the "package management" approach you should gradually (one at a time) split the mono repo in separate repos, and enable the decoupled development. If you are using a mono-repo approach, then you might not need a package manager in the first place?

[memsharded]

I am investigating in conan-io/conan#13562 some possible changes that could allow the revision_mode = "scm_xxxx" (where xxx could be something like scm_subfolder, for example) to capture the commit of the subfolder. Still, the above rationale and concerns about the overall approach holds.

[xiaoliangwu]

The pain point is: in developer's sandbox , Make can save the compile time if the .o or .a exist by checking the timestamp. but in CICD which using a clean workspace, everything need to be compiled and it takes a long time, so I think Conan can help to skip modules by checking the source revision and speed up the build TAT in CI pipeline.

thanks for the PR for scm_subfolder approach , that could be helpful .

for a more pragmatic approach, maybe we can add something like "recipe_revision()" to let user decide the revision. however, this is just my 2cents

[memsharded]

The pain point is: in developer's sandbox , Make can save the compile time if the .o or .a exist by checking the timestamp. but in CICD which using a clean workspace, everything need to be compiled and it takes a long time, so I think Conan can help to skip modules by checking the source revision and speed up the build TAT in CI pipeline.

I love that you are trying Conan for this, but maybe if the only pain is build time, have you considered using a build cache (ccache, scache)?

If you still want to stick with Conan for this for other reasons, I think that the exports_sources approach with the standard revision_mode = "hash" (default, don't need to be specified) will give you the behavior that you want in CI, without needing to deal with explicit revisions in your conanfiles or in your requirements at all. Something like this:

• Use a custom command like https://github.com/conan-io/conan-extensions/blob/main/extensions/commands/cci/cmd_export_all_versions.py to export all recipes in your mono repo, with their exports_sources. A simple script might also be good.
• Do the conan create or conan install commands to build things. As the recipes with the recipe revision that are in the cache have been just exported, Conan will be resolving to them, as they are already the latest. No need to explicitly add that revision in the requirements()
• Use the correct package_id modes (Conan 2.0 has better defaults than 1.X), and let --build=missing argument do the job. It will skip what is not necessary to build, and will retrieve from the server existing binaries that don't need to be rebuilt
• A root conanfile_project.py that requires the important, first level subprojects in the repo can help orchestrating the build in an easier way.
• At the end, upload all the new binaries to the server for next build.

for a more pragmatic approach, maybe we can add something like "recipe_revision()" to let user decide the revision. however, this is just my 2cents

This would not be necessary, the recipes already have the set_version() method. If you want to define a custom revision, it is as simple as appending an extra digit to the version, for example. A revision is nothing but an automatic extra version item, that is sorted by timestamp and implicitly resolved to latest. If you control and can generate your own revision (it is not as simple as it seems to do it in a decentralized way, actually it can be very challenging), then append it to the version (you can use +build.Number for example in 2.0, as they are sorted and resolved too).

[xiaoliangwu]

I like the "exports_sources" approach, perfect! BTW, what is conafile_project.py? is it a special recipe file superior to conanfile.py?

[memsharded]

No, sorry for the confusion conanfile_project.py was just a way of trying to explain it. It is true that you can name a conanfile differently and then use it like conan install conanfile_project.py, but that is not very typical.

A conanfile.py in the root of the project or something, that serves as the entry point for the whole project (it can have dependencies requires to the most bottom subprojects, like the consuming applications, and that way it is easy to build a full dependency graph for the whole project)

[memsharded]

conan-io/conan#13562 implemented revision_mode = "scm_folder" (for next 2.0.5) that will pick the git commit of the folder containing the conanfile.py, so that would address your issue if you still want to do that.

Closing this issue, please create a new one if you have any further question. Thanks for the feedback!