import: Git SSH keyfile auth broken in dvc>=2.44.0

[ReveStobinson]

Bug Report

Description

dvc import fails to authenticate to the specified Git remote with an SSH keyfile (In this case, a GitLab repository on a locally hosted private instance).

The following output is produced:

Importing 'path/to/data/ (git@internal.gitlab.url:path/to/repo.git)' -> 'rawdata'
ERROR: unexpected error - Git failed to fetch ref from 'git@internal.gitlab.url:path/to/repo.git': authentication required but no callback set   

Reproduce

The nonfunctioning command is this:

dvc import git@internal.gitlab.url:path/to/repo.git \
  path/to/data \
  -o rawdata/

The directory to be downloaded is located in a .dvc-initialized subdirectory within the remote repository in question. That location has in it's .dvc/config the default remote pointing to the ssh path to our dvc remote storage (On a different machine from the one hosting the GitLab instance).

Prior to release 2.44.0, as long as we had our SSH config set up properly and used an SSH keyfile that did not require a password, this command would successfully download the files. Now, the above error is always encountered with any DVC version newer than 2.44.0.

Downgrading to 2.43.4 or lower resolves the issue.

Expected

dvc import to download the files to the ./rawdata directory, and begin tracking them with DVC (add rawdata.dvc file, add /rawdata to .gitignore, etc.).

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.44.0 (pip)
-------------------------
Platform: Python 3.9.16 on Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 0.39.3
        dvc_objects = 0.19.3
        dvc_render = 0.1.1
        dvc_task = 0.1.11
        dvclive = 2.0.2
        scmrepo = 0.1.9
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.1.0)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sdc
Repo: dvc, git

Other potential dependencies of relevance:

dvc-http                  2.30.2
dvc-ssh                   2.21.0
dvc-studio-client         0.5.0

asyncssh             2.13.1
sshfs              2023.1.0

gitdb                4.0.10
gitpython            3.1.31
pygit2               1.11.1

Additional Information (if any):

Additional error info from --verbose:

File "~/mambaforge/envs/dvc_2.44.0/lib/python3.9/site-packages/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: authentication required but no callback set

Working environment:

I am able to produce the expected output with the same command in this environment:

DVC version: 2.43.4 (pip)
-------------------------
Platform: Python 3.9.16 on Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 0.37.8
        dvc_objects = 0.19.3
        dvc_render = 0.1.0
        dvc_task = 0.1.11
        dvclive = 2.0.2
        scmrepo = 0.1.7
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.1.0)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sdc
Repo: dvc, git

Additional dependencies: (All the same as non-working env)

dvc-http                  2.30.2
dvc-ssh                   2.21.0
dvc-studio-client         0.5.0

asyncssh             2.13.1
sshfs              2023.1.0

gitdb                4.0.10
gitpython            3.1.31
pygit2               1.11.1

[pmrowla]

Can you try updating to the latest release (2.45.1)? This is likely the same underlying bug as #9016

[petrchmelar]

There is a possibility that the import is not going to work with 2.45.1 also. As I mentioned in #9016 (comment) the dvc is still failing to import or just list the repository over ssh when no credentials helper configured.

[gregstarr]

dvc exp pull origin --rev <sha> did not work for me in 2.45.1 but does work in 2.43.4

[daavoo]

Regression from #9041 / iterative/scmrepo#189

[ReveStobinson]

@pmrowla

Can you try updating to the latest release (2.45.1)? This is likely the same underlying bug as #9016

My team actually first noticed this issue in version 2.45.1. I went back through the release history afterwards and found the issue in all versions from 2.44.0 and up (did not test any pre-release versions).

[ReveStobinson]

Just manually installed the 0.1.13 release of scmrepo created this morning from the PRs this issue was mentioned in (with dvc==2.45.1), and it appears to resolve this issue! Once a stable release of dvc has scmrepo>=0.1.13 as a dependency, this issue can be closed.

[exs-dmiketa]

Installing scmrepo==0.1.13 unfortunately didn't fix this issue for me on dvc==2.45.1 (using Bitbucket).

[shcheklein]

@iterative/dvc since it was a p0 do we want to do a postmortem, any lessons from this?

[daavoo]

Installing scmrepo==0.1.13 unfortunately didn't fix this issue for me on dvc==2.45.1 (using Bitbucket).

Can confirm that the issue still exists for exp pull when the git remote points to an SSH URL.
Working on a fix

[daavoo]

https://github.com/orgs/iterative/teams/dvc since it was a p0 do we want to do a postmortem, any lessons from this?

will do

[daavoo]

@iterative/dvc since it was a p0 do we want to do a postmortem, any lessons from this?

#9117