[question] Migration to 2.0 - SCM removal

[MichalCermakx]

What is your question?

Hi,

I have a question about the removal of SCM. I was migrating my conanfiles to 2.0 using this guide: https://docs.conan.io/en/latest/migrating_to_2.0/recipes.html#the-scm-attribute
Overall it works fine, but there is one difference. When I have the repository already cloned with conanfile in it, the 1.x way using SMC copies this repository content to conan_cache/source_folder.
With the update to 2.0 source, I am getting a call to git clone, which was not happening with 1.x.

Is this expected? Is there a way to mimic the old behavior?

Thanks,
Michal

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @MichalCermakx

Thanks for your question.

Is this expected? Is there a way to mimic the old behavior?

Yes, it is expected. The new logic is more robust and consistent than the old one, that had several important flaws (like copying sources from user folders, outdated sources, after the package has been already created in the cache and uploaded).

The changes were done to guarantee that the scm based builds are reproducible, clean and complete. It is indeed doing a git clone instead of a folder copy, it might have the inconvenience to be a bit slower (maybe it can be optimized with --depth=1 or something like that), but the "copying" approach was too problematic, so it is not expected to be possible.

[MichalCermakx]

Hi @memsharded ,

Your explanation makes sense and I understand the benefits of cloning the repository again.

Let me add reasons why I like the 1.x behavior.

1. I am using Bamboo for creating Conan packages from GIT. I am not sure if other CI solutions have this limit, but Bamboo has a specific task for git clone and other git interactions (https://confluence.atlassian.com/bamboo/checking-out-code-289277060.html). In other words, you are not calling git clone via commandline. This solves the authentication of a build agent (SSH key exchange) behind the scene. When you need to interact with git via CLI, you need to use http GIT authentication (Not ideal), or somehow deploy a custom SSH key to build agent.
2. Added cost of cloning. We will end with 2 clones, one used during the export(), and the second in the source(). You are suggesting to use --depth=1. This is not as easy, because this is related to a specific branch (https://git-scm.com/docs/git-clone), but in this case, we need SHA.
   I found some workarounds (https://stackoverflow.com/questions/3489173/how-to-clone-git-repository-with-specific-revision-changeset), but those do not use "git clone", but rather some sequence of git commands (init, remote add, fetch SHA). So sadly, the Git.clone implementation in Conan is not usable. Workaround is to run those commands via Git.run sequence.

At high-level, I understand this change, but I will miss the simplicity of a Conan 1.x approach, which does not lead to the issues described above.

Thanks.

[memsharded]

Thanks for the explanation.

However I'd like to understand a bit better the first point:

I am using Bamboo for creating Conan packages from GIT. I am not sure if other CI solutions have this limit, but Bamboo has a specific task for git clone and other git interactions (https://confluence.atlassian.com/bamboo/checking-out-code-289277060.html). In other words, you are not calling git clone via commandline. This solves the authentication of a build agent (SSH key exchange) behind the scene. When you need to interact with git via CLI, you need to use http GIT authentication (Not ideal), or somehow deploy a custom SSH key to build agent.

The git clone need to happen anyway at some point, with the legacy scm feature, because this clone is still happening whenever it is necessary to build from source, other than a conan create (like a conan install --build=misssing, because the pre-compiled binary is missing for a configuration). So whatever needs to be set up regarding Git, using https, credentials, etc, should be already configured anyway, shouldn't it?

Regarding 2, if that sequence is of value, and it could be useful for other users as well, then we could definitely make it built-in in Git, its design is now cleaner, so I wouldn't oppose to add this kind of functionality there.

[MichalCermakx]

Thanks @memsharded ,

You are absolutely right, that when the binary package is missing, a rebuild of a package will happen. But in my environment, it is not happening (At least not now). I am prebuilding the binary variants we need and storing them in my company's Artifactory.
Anyway, I will try to find a way to handle this scenario, because it seems to be more like an issue on Bamboo side rather than on Conan.

Regarding the shallow copy, this code words for me:

    git_shallow_clone = True

    def source(self):
        git = Git(self, self.git_repo_source_folder)
        sources = self.conan_data["sources"]
        if self.git_shallow_clone:
            git.run('init')
            git.run(f'remote add origin {sources["url"]}')
            git.run(f'fetch --depth 1 origin {sources["commit"]}')
            git.run(f'checkout FETCH_HEAD')
        else:
            git.clone(url=sources["url"], target='.')
            git.checkout(commit=sources["commit"])

Feel free to use this code in Conan. Note: as described in stackowerflow link, it needs to have at least some minimum version of a git on server side. Therefore it might not work for everybody.

[memsharded]

Thanks for sharing!
I have done #13096 following your code, to implement the "fetch_commit", I hope that will help to maintain recipes a bit cleaner

[memsharded]

This is strange: #13096 only fails in Linux, but I can't find the reason. I keep getting:

    CalledProcessErrorWithStderr: Command 'git fetch --depth 1 origin 670fdc342d78212aff1786aec96f0a05ba06cdd7' returned non-zero exit status 1.
E           error: Server does not allow request for unadvertised object 670fdc342d78212aff1786aec96f0a05ba06cdd7

Do you use that approach in Linux too? Have you seen this before?

[MichalCermakx]

If I got it, it needs to be supported by git on Server side (Check this comment - https://stackoverflow.com/a/3489576)
I have tested 3 public repos in GitLab, GitHub, and bitbucket, and all were working fine. Therefore I believe your git server uses old git version. Is it possible?

[memsharded]

I have tested 3 public repos in GitLab, GitHub, and bitbucket, and all were working fine. Therefore I believe your git server uses old git version. Is it possible?

It is not even a server, it is a clone directly from a local folder. But yes, it is likely that the version and conf in Linux of git might be different.

[memsharded]

#13096 merged for 1.59