scm: use dulwich backend when fetching exps during clone/pull

[skshetry]

refspecs uses pygit2 backend by default, which does not support git credential helpers. This PR forces to use dulwich backend during clone/pull operations. Other uses of refspecs (in experiments in particular) is broken for users of git credential-helpers though.

Fixes #9016.

[codecov]

Codecov Report

Base: 93.07% // Head: 93.06% // Decreases project coverage by -0.02% ⚠️

Coverage data is based on head (683643d) compared to base (eff775c).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9023      +/-   ##
==========================================
- Coverage   93.07%   93.06%   -0.02%     
==========================================
  Files         455      455              
  Lines       36753    36753              
  Branches     5316     5316              
==========================================
- Hits        34208    34203       -5     
- Misses       2023     2026       +3     
- Partials      522      524       +2     

Impacted Files	Coverage Δ
dvc/external_repo.py	81.81% <100.00%> (ø)
dvc/repo/experiments/utils.py	82.08% <100.00%> (ø)
dvc/scm.py	86.42% <100.00%> (ø)
dvc/repo/experiments/queue/celery.py	85.92% <0.00%> (-1.86%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[daavoo]

(Not blocking this) Should we set up some tests using credential manager?

[skshetry]

We don't run any dockerized tests in dvc at the moment. We could test in scmrepo, but the whole exp-refs/external_repo logic remains in dvc.

[skshetry]

I can take a look though.

[pmrowla]

Seems like this should be resolved on the scmrepo end. We can move credentials.py up a level into scmrepo.git since the generic logic for checking if there is a configured credential for a given git URL is not dulwich specific. And from there we just have to check if there is a configured credentialstore for our remote URL in the pygit backend before doing fetch/push, and raise NotImplementedError when there is one.

(and from there DVC doesn't need to do anything, the automatic backend stuff will try pygit first and then fall back to dulwich when it has to)

[skshetry]

Decided not to add, I think this should be natively supported by our backends. We cannot and should not test every kind of configuration.

[skshetry]

just have to check if there is a configured credentialstore for our remote URL in the pygit backend

if we do that, we should just add support for git credential-helpers.

[pmrowla]

just have to check if there is a configured credentialstore for our remote URL in the pygit backend

if we do that, we should just add support for git credential-helpers.

We already have generic logic for doing this checking the configuration as it only requires reading .git/config which is supported in every backend. Actually using the credential helper in pygit requires adding support for it in libgit2, which I would consider non-trivial as it will require writing C

[skshetry]

I'd just prioritize having parity between different backends instead of using complicated logic to choose backends. Implementing git credential-helpers will offer the same performance.

[skshetry]

Actually using the credential helper in pygit requires adding support for it in libgit2, which I would consider non-trivial as it will require writing C

I looked into it briefly, and it seems we can implement it by extending pygit2.RemoteCallbacks.credentials().

[pmrowla]

Actually, looking at what's in credentials.py we can probably just reuse what the dulwich backend is already doing in the scmrepo pygit wrapper instead of needing to add upstream support

[skshetry]

Yes, we can reuse most of the stuff, we need to interface with RemoteCallbacks.credentials() and return pygit2.UserPass() from it.

[dberenbaum]

@skshetry Thank you for the fix! Can you open a follow-up issue for adding credential helpers in pygit2?

[skshetry]

iterative/scmrepo#190