dvc import broken authentication #7898

wdixon · 2022-06-14T16:12:40Z

We seem to no longer be able to import a remote dvc resources from a repository that requires authentication. This worked at some point prior to 2.10. This worked previously.

The output of the import is as follows:

2022-06-14 12:01:24,323 ERROR: failed to import 'model.hdf5' from 'https://github.build.company.com/org/Artifacts.git'. - Failed to clone repo 'https://github.build.company.com/org/Artifacts.git' to '/var/tmp/pbs.1540626.scinfra2/tmpc9sse3s9dvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/backend/dulwich/__init__.py", line 196, in clone
    repo = clone_from()
  File "/usr/local/lib/python3.8/dist-packages/dulwich/porcelain.py", line 443, in clone
    return client.clone(
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 747, in clone
    result = self.fetch(path, target, progress=progress, depth=depth)
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 824, in fetch
    result = self.fetch_pack(
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2079, in fetch_pack
    refs, server_capabilities, url = self._discover_references(
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 1938, in _discover_references
    resp, read = self._http_request(url, headers)
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2219, in _http_request
    raise HTTPUnauthorized(resp.getheader("WWW-Authenticate"), url)
dulwich.client.HTTPUnauthorized: No valid credentials provided

The above exception was the direct cause of the following exception:

There was some previous commit appears to fix a similar issue, which was part of the 2.11 release:
issue #7670
however, updating to 2.11 did not fix the issue.

DVC version: 2.11.0 (pip)
---------------------------------
Platform: Python 3.8.0 on Linux-3.10.0-1160.66.1.el7.x86_64-x86_64-with-glibc2.27
Supports:
	webhdfs (fsspec = 2022.5.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2022.5.0, boto3 = 1.21.21)
Cache types: hardlink, symlink
Cache directory: nfs on LEB1MLNAS.hpc.company.com:/leb1mlnas_projects
Caches: local
Remotes: None
Workspace directory: nfs on LEB1MLNAS.hpc.company.com:/leb1mlnas_projects
Repo: dvc, git

efiop · 2022-06-14T16:24:20Z

CC @dtrifiro

wdixon · 2022-06-14T20:44:38Z

I reverted to version 2.9.5 and can now perform the dvc import of a remote resource as expected.

I would also point out that dvc update was also not working - failing for the same authentication issue.

dtrifiro · 2022-06-16T08:51:29Z

Hey @wdixon, would you mind providing a full traceback? Thanks

wdixon · 2022-06-16T10:33:14Z

I don't have a configuration that I can run from the same system.... but here is the same issue run from a windows system (the traceback doesn't appear to be as verbose - but the line numbers seem to match the traceback from the earlier linux system). Does this have the info you need?

$ dvc doctor
DVC version: 2.11.0 (exe)
---------------------------------
Platform: Python 3.8.10 on Windows-10-10.0.19042-SP0
Supports:
        azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.10.0),
        gdrive (pydrive2 = 1.10.0),
        gs (gcsfs = 2021.10.1),
        hdfs (fsspec = 2021.10.1, pyarrow = 6.0.0),
        webhdfs (fsspec = 2021.10.1),
        http (aiohttp = 3.8.0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.0, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2021.10.1, boto3 = 1.17.106),
        ssh (sshfs = 2021.11.2),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.3),
        webdavs (webdav4 = 0.9.3)
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git

$ dvc import --verbose https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git pose/S1FV_PS_A_20220606-092610-0558-0.0003.hdf5
2022-06-16 11:22:48,163 DEBUG: Removing output 'S1FV_PS_A_20220606-092610-0558-0.0003.hdf5' of stage: 'S1FV_PS_A_20220606-092610-0558-0.0003.hdf5.dvc'.
2022-06-16 11:22:48,163 DEBUG: Removing 'C:\cygwin64\home\200003581\onwing_s1b_keyframe\onwing_s1b_keyframe\ckpts\tmp\S1FV_PS_A_20220606-092610-0558-0.0003.hdf5'
Importing 'pose/S1FV_PS_A_20220606-092610-0558-0.0003.hdf5 (https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git)' -> 'S1FV_PS_A_20220606-092610-0558-0.0003.hdf5'
2022-06-16 11:22:48,173 DEBUG: Computed stage: 'S1FV_PS_A_20220606-092610-0558-0.0003.hdf5.dvc' md5: 'fb7f834659a494bd33f117228c03a0d7'
2022-06-16 11:22:48,173 DEBUG: 'md5' of stage: 'S1FV_PS_A_20220606-092610-0558-0.0003.hdf5.dvc' changed.
2022-06-16 11:22:48,173 DEBUG: Creating external repo https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git@None
2022-06-16 11:22:48,173 DEBUG: erepo: git clone 'https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git' to a temporary dir
2022-06-16 11:22:48,674 ERROR: failed to import 'pose/S1FV_PS_A_20220606-092610-0558-0.0003.hdf5' from 'https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git'. - Failed to clone repo 'https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git' to 'C:\cygwin64\tmp\tmp8zsh7ht1dvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
  File "scmrepo\git\backend\dulwich\__init__.py", line 196, in clone
  File "dulwich\porcelain.py", line 443, in clone
  File "dulwich\client.py", line 747, in clone
  File "dulwich\client.py", line 824, in fetch
  File "dulwich\client.py", line 2079, in fetch_pack
  File "dulwich\client.py", line 1938, in _discover_references
  File "dulwich\client.py", line 2219, in _http_request
dulwich.client.HTTPUnauthorized: No valid credentials provided

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc\scm.py", line 145, in clone
  File "scmrepo\git\__init__.py", line 143, in clone
  File "scmrepo\git\backend\dulwich\__init__.py", line 199, in clone
scmrepo.exceptions.CloneError: Failed to clone repo 'https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git' to 'C:\cygwin64\tmp\tmp8zsh7ht1dvc-clone'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc\commands\imp.py", line 15, in run
  File "dvc\repo\imp.py", line 6, in imp
  File "dvc\repo\__init__.py", line 49, in wrapper
  File "dvc\repo\scm_context.py", line 152, in run
  File "dvc\repo\imp_url.py", line 83, in imp_url
  File "funcy\decorators.py", line 45, in wrapper
  File "dvc\stage\decorators.py", line 36, in rwlocked
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\stage\__init__.py", line 535, in run
  File "funcy\decorators.py", line 45, in wrapper
  File "dvc\stage\decorators.py", line 36, in rwlocked
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\stage\__init__.py", line 559, in _sync_import
  File "dvc\stage\imports.py", line 47, in sync_import
  File "dvc\dependency\repo.py", line 68, in download
  File "dvc\dependency\repo.py", line 97, in get_used_objs
  File "dvc\dependency\repo.py", line 111, in _get_used_and_obj
  File "contextlib.py", line 113, in __enter__
  File "dvc\external_repo.py", line 39, in external_repo
  File "dvc\external_repo.py", line 169, in _cached_clone
  File "funcy\decorators.py", line 45, in wrapper
  File "funcy\flow.py", line 274, in wrap_with
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\external_repo.py", line 239, in _clone_default_branch
  File "dvc\scm.py", line 150, in clone
dvc.scm.CloneError: Failed to clone repo 'https://github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git' to 'C:\cygwin64\tmp\tmp8zsh7ht1dvc-clone'
------------------------------------------------------------
2022-06-16 11:22:48,689 DEBUG: Analytics is enabled.
2022-06-16 11:22:48,689 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\cygwin64\\tmp\\tmp13dedzle']'
2022-06-16 11:22:48,705 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\cygwin64\\tmp\\tmp13dedzle']'

dtrifiro · 2022-06-16T11:30:42Z

Hi @wdixon, yes. Thank you.

This is a known issue (e.g. see #7670, #6586)

The problem is that since 2.9.5, scmrepo was bumped and in 0.0.11 the dulwich git backend for clone was implemented (favoring that over GitPython). Note that GitPython uses CLI git under the hood (which is not always available, hence the move to dulwich).

I'm assuming the http auth credentials are being passed to git using a credential helper defined in your git configuration.

Clone breaks because dulwich has no way of providing credentials for authentication since it doesn't support credential helpers (yet).

If you wish to use dvc > 2.9.5 the workaround for http auth is providing username/password in the URL you wish to import:

dvc import https://username:password@github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git pose/S1FV_PS_A_20220606-092610-0558-0.0003.hdf5

The good news is that there's an open PR (jelmer/dulwich#976) which adds support for credential helpers. If you want to test it out, you can try installing it:

pip install git+https://github.com/dtrifiro/dulwich.git@feature/credential-helper

Any feedback would be appreciated!

wdixon · 2022-06-16T11:44:35Z

Our backend enterprise git is configured to disallow username:password in the URL; however, it does accept tokens in the URL.... But unfortunately this doesn't seem to work work either - though this would be a more common way to pass credentials that including the password in the clear....

Using the token in the URL produces a different exception (see below).

Traceback (most recent call last):
  File "scmrepo\git\backend\dulwich\__init__.py", line 196, in clone
  File "dulwich\porcelain.py", line 443, in clone
  File "dulwich\client.py", line 747, in clone
  File "dulwich\client.py", line 824, in fetch
  File "dulwich\client.py", line 2079, in fetch_pack
  File "dulwich\client.py", line 1938, in _discover_references
  File "dulwich\client.py", line 2223, in _http_request
dulwich.errors.GitProtocolError: unexpected http resp 403 for https://0b9ad3ae_rest_of_token@github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git/info/refs?service=git-upload-pack

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc\scm.py", line 145, in clone
  File "scmrepo\git\__init__.py", line 143, in clone
  File "scmrepo\git\backend\dulwich\__init__.py", line 199, in clone
scmrepo.exceptions.CloneError: Failed to clone repo 'https://0b9ad3ae_rest_of_token@github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git' to 'C:\cygwin64\tmp\tmp2vfmbeqjdvc-clone'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc\commands\imp.py", line 15, in run
  File "dvc\repo\imp.py", line 6, in imp
  File "dvc\repo\__init__.py", line 49, in wrapper
  File "dvc\repo\scm_context.py", line 152, in run
  File "dvc\repo\imp_url.py", line 83, in imp_url
  File "funcy\decorators.py", line 45, in wrapper
  File "dvc\stage\decorators.py", line 36, in rwlocked
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\stage\__init__.py", line 535, in run
  File "funcy\decorators.py", line 45, in wrapper
  File "dvc\stage\decorators.py", line 36, in rwlocked
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\stage\__init__.py", line 559, in _sync_import
  File "dvc\stage\imports.py", line 47, in sync_import
  File "dvc\dependency\repo.py", line 68, in download
  File "dvc\dependency\repo.py", line 97, in get_used_objs
  File "dvc\dependency\repo.py", line 111, in _get_used_and_obj
  File "contextlib.py", line 113, in __enter__
  File "dvc\external_repo.py", line 39, in external_repo
  File "dvc\external_repo.py", line 169, in _cached_clone
  File "funcy\decorators.py", line 45, in wrapper
  File "funcy\flow.py", line 274, in wrap_with
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\external_repo.py", line 239, in _clone_default_branch
  File "dvc\scm.py", line 150, in clone
dvc.scm.CloneError: Failed to clone repo 'https://0b9ad3ae_rest_of_token@github.build.company.com/auto-inspection/BIT_Tool_Artifacts.git' to 'C:\cygwin64\tmp\tmp2vfmbeqjdvc-clone'
------------------------------------------------------------
2022-06-16 12:37:28,104 DEBUG: Analytics is enabled.
2022-06-16 12:37:28,107 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\cygwin64\\tmp\\tmp4cmxuv_z']'
2022-06-16 12:37:28,117 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\cygwin64\\tmp\\tmp4cmxuv_z']'

wdixon · 2022-06-20T16:21:01Z

The token does comply with the RFC.... It is simply the case where no password is provided, only the user (as the token)

Directly from the RFC text:

        //<user>:<password>@<host>:<port>/<url-path>

password
        An optional password. If present, it follows the user
        name separated from it by a colon.

When authenticating using basic auth and only providing username (no password) the default value of `None` was cast to string, resulting in an attempt to authenticate using `<username>:None` instead of providing an empty password. This broke authentication when providing an auth token as username. See iterative/dvc#7898 (comment)

dtrifiro · 2022-06-20T18:07:38Z

I'm sorry, you're right. That is a bug in dulwich. Opened up a PR upstream. See jelmer/dulwich#980

dtrifiro · 2022-06-21T11:15:48Z

Recap:

For broken imports related to the change from gitpython backend to dulwich: we're waiting for git credentials support in dulwich (add support for git credential helpers jelmer/dulwich#976)
For broken https://token@host... authentication: it has been merged in dulwich, so we're waiting a release and scmrepo version bump

dtrifiro · 2022-07-11T18:13:47Z

@wdixon dulwich has merged the basic auth fix, so by using dulwich >= 0.20.44 you should be able to use dvc import with the token@host syntax. Still waiting on proper git credentials support though

d-miketa · 2022-07-27T21:49:21Z

@dtrifiro FWIW I tried installing dulwich from your feature branch using

pip install git+https://github.com/dtrifiro/dulwich.git@feature/credential-helper

and I'm unfortunately still running into the same issue:

❯ dvc pull data/dataset.dvc
Everything is up to date.                                                                                                                                       
ERROR: failed to pull data from the cloud - Failed to clone repo 'git@bitbucket.org:organisation/datasets.git' to '/var/folders/b1/d587c0_d6hb6ndr4mv24h_gc0000gq/T/tmpu75hs20xdvc-clone'

Downgrading to 2.9 fixes the issue. I'm on macOS.

dtrifiro · 2022-07-28T10:23:51Z

@d-miketa thanks! Could you post a full traceback using dvc pull -v?

d-miketa · 2022-07-28T16:48:26Z

I got rid of the import but here's a similar issue when trying to do dvc exp list origin; also crops up when pushing an experiment:

❯ dvc exp list origin -v
2022-07-28 17:44:28,077 ERROR: unexpected error - Permission denied   
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/cli/__init__.py", line 185, in main
    ret = cmd.do_run()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/commands/experiments/ls.py", line 13, in run
    exps = self.repo.experiments.ls(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 539, in ls
    return ls(self.repo, *args, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 156, in run
    return method(repo, *args, **kw)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/experiments/ls.py", line 34, in ls
    ref_info_dict = exp_refs_by_baseline(repo.scm, rev_set, git_remote)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/experiments/utils.py", line 87, in exp_refs_by_baseline
    for ref in all_exp_refs:
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/experiments/utils.py", line 68, in exp_refs
    for ref in ref_gen:
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/experiments/utils.py", line 101, in iter_remote_refs
    yield from scm.iter_remote_refs(url, base=base, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 483, in iter_remote_refs
    for ref in client.get_refs(path)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dulwich/client.py", line 1170, in get_refs
    proto, _, stderr = self._connect(b"upload-pack", path)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dulwich/client.py", line 1790, in _connect
    con = self.ssh_vendor.run_command(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/fsspec/asyn.py", line 86, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/fsspec/asyn.py", line 66, in sync
    raise return_result
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/fsspec/asyn.py", line 26, in _runner
    result[0] = await coro
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 163, in _run_command
    conn = await asyncssh.connect(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/asyncssh/connection.py", line 6892, in connect
    return await asyncio.wait_for(
  File "/usr/local/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/asyncssh/connection.py", line 303, in _connect
    await conn.wait_established()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/asyncssh/connection.py", line 2262, in wait_established
    await self._waiter
asyncssh.misc.PermissionDenied: Permission denied
------------------------------------------------------------
2022-07-28 17:44:28,323 DEBUG: Removing '/Users/dmiketa/repos/.MZNyYcsLnqwbapzFKucrCz.tmp'
2022-07-28 17:44:28,324 DEBUG: Removing '/Users/dmiketa/repos/.MZNyYcsLnqwbapzFKucrCz.tmp'
2022-07-28 17:44:28,324 DEBUG: Removing '/Users/dmiketa/repos/.MZNyYcsLnqwbapzFKucrCz.tmp'
2022-07-28 17:44:28,325 DEBUG: Removing '/Users/dmiketa/repos/my-repo/.dvc/cache/.7N7f3x45tqJjjTxHRLe2K7.tmp'
2022-07-28 17:44:28,327 DEBUG: Version info for developers:
DVC version: 2.15.0 (pip)
---------------------------------
Platform: Python 3.10.5 on macOS-12.4-x86_64-i386-64bit
Supports:
	webhdfs (fsspec = 2022.5.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
	s3 (s3fs = 2022.5.0, boto3 = 1.21.21)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git

d-miketa · 2022-07-28T16:54:37Z

And for the dvc pull -v you get

2022-07-28 17:51:34,830 ERROR: failed to import 'data/formatted' from 'git@bitbucket.org:organisation/datasets.git'. - Failed to clone repo 'git@bitbucket.org:organisation/datasets.git' to '/var/folders/b1/d587c0_d6hb6ndr4mv24h_gc0000gq/T/tmpc1a2blntdvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 196, in clone
    repo = clone_from()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dulwich/porcelain.py", line 443, in clone
    return client.clone(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dulwich/client.py", line 750, in clone
    result = self.fetch(path, target, progress=progress, depth=depth)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dulwich/client.py", line 827, in fetch
    result = self.fetch_pack(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dulwich/client.py", line 1121, in fetch_pack
    proto, can_read, stderr = self._connect(b"upload-pack", path)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dulwich/client.py", line 1790, in _connect
    con = self.ssh_vendor.run_command(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/fsspec/asyn.py", line 86, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/fsspec/asyn.py", line 66, in sync
    raise return_result
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/fsspec/asyn.py", line 26, in _runner
    result[0] = await coro
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 163, in _run_command
    conn = await asyncssh.connect(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/asyncssh/connection.py", line 6892, in connect
    return await asyncio.wait_for(
  File "/usr/local/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/asyncssh/connection.py", line 303, in _connect
    await conn.wait_established()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/asyncssh/connection.py", line 2262, in wait_established
    await self._waiter
asyncssh.misc.PermissionDenied: Permission denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/scm.py", line 145, in clone
    git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 143, in clone
    backend.clone(url, to_path, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 199, in clone
    raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'git@bitbucket.org:organisation/datasets.git' to '/var/folders/b1/d587c0_d6hb6ndr4mv24h_gc0000gq/T/tmpc1a2blntdvc-clone'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/commands/imp.py", line 15, in run
    self.repo.imp(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/imp.py", line 6, in imp
    return self.imp_url(
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 156, in run
    return method(repo, *args, **kw)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/repo/imp_url.py", line 83, in imp_url
    stage.run(jobs=jobs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/stage/__init__.py", line 535, in run
    self._sync_import(dry, force, jobs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/stage/__init__.py", line 559, in _sync_import
    sync_import(self, dry, force, jobs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/stage/imports.py", line 43, in sync_import
    stage.deps[0].download(stage.outs[0], jobs=jobs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/dependency/repo.py", line 68, in download
    for odb, objs in self.get_used_objs().items():
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/dependency/repo.py", line 97, in get_used_objs
    used, _ = self._get_used_and_obj(**kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/dependency/repo.py", line 111, in _get_used_and_obj
    with self._make_repo(
  File "/usr/local/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/external_repo.py", line 39, in external_repo
    path = _cached_clone(url, rev, for_write=for_write)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/external_repo.py", line 169, in _cached_clone
    clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/funcy/flow.py", line 274, in wrap_with
    return call()
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/external_repo.py", line 239, in _clone_default_branch
    git = clone(url, clone_path)
  File "/Users/dmiketa/.local/pipx/venvs/dvc/lib/python3.10/site-packages/dvc/scm.py", line 150, in clone
    raise CloneError(str(exc))
dvc.scm.CloneError: Failed to clone repo 'git@bitbucket.org:organisation/datasets.git' to '/var/folders/b1/d587c0_d6hb6ndr4mv24h_gc0000gq/T/tmpc1a2blntdvc-clone'

dtrifiro · 2022-08-02T16:00:05Z

Hey @d-miketa, thanks for the feedback. If you test out the same branch, it should be working now

d-miketa · 2022-08-02T19:37:06Z

Thanks @dtrifiro, that did the trick!! :)

dtrifiro · 2022-09-13T15:03:07Z

Solved in 2.25.0

dtrifiro self-assigned this Jun 15, 2022

dtrifiro added this to DVC Jun 16, 2022

dtrifiro moved this to Backlog in DVC Jun 16, 2022

dtrifiro moved this from Backlog to In Progress in DVC Jun 16, 2022

dtrifiro added regression Ohh, we broke something :-( A: data-sync Related to dvc get/fetch/import/pull/push labels Jun 16, 2022

This comment was marked as outdated.

Sign in to view

dtrifiro mentioned this issue Jun 20, 2022

client: fix basic auth with empty password jelmer/dulwich#980

Merged

dtrifiro closed this as completed Sep 13, 2022

Repository owner moved this from In Progress to Done in DVC Sep 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dvc import broken authentication #7898

dvc import broken authentication #7898

wdixon commented Jun 14, 2022

efiop commented Jun 14, 2022

wdixon commented Jun 14, 2022

dtrifiro commented Jun 16, 2022

wdixon commented Jun 16, 2022

dtrifiro commented Jun 16, 2022

wdixon commented Jun 16, 2022

This comment was marked as outdated.

wdixon commented Jun 20, 2022

dtrifiro commented Jun 20, 2022

dtrifiro commented Jun 21, 2022

dtrifiro commented Jul 11, 2022

d-miketa commented Jul 27, 2022

dtrifiro commented Jul 28, 2022

d-miketa commented Jul 28, 2022

d-miketa commented Jul 28, 2022

dtrifiro commented Aug 2, 2022

d-miketa commented Aug 2, 2022

dtrifiro commented Sep 13, 2022

dvc import broken authentication #7898

dvc import broken authentication #7898

Comments

wdixon commented Jun 14, 2022

efiop commented Jun 14, 2022

wdixon commented Jun 14, 2022

dtrifiro commented Jun 16, 2022

wdixon commented Jun 16, 2022

dtrifiro commented Jun 16, 2022

wdixon commented Jun 16, 2022

This comment was marked as outdated.

wdixon commented Jun 20, 2022

dtrifiro commented Jun 20, 2022

dtrifiro commented Jun 21, 2022

dtrifiro commented Jul 11, 2022

d-miketa commented Jul 27, 2022

dtrifiro commented Jul 28, 2022

d-miketa commented Jul 28, 2022

d-miketa commented Jul 28, 2022

dtrifiro commented Aug 2, 2022

d-miketa commented Aug 2, 2022

dtrifiro commented Sep 13, 2022