DVC list FileNotFoundError #7120

nikshar-symbio · 2021-12-09T20:51:43Z

Bug Report

`dvc list . : gives "ERROR: unexpected error"`

Description

When trying to list the files using dvc list ., I get the following error

2021-12-01 11:32:18,645 ERROR: unexpected error
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/command/base.py", line 64, in do_run
    return self.run()
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/command/ls/__init__.py", line 30, in run
    entries = Repo.ls(
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/repo/ls.py", line 36, in ls
    ret = _ls(repo.repo_fs, path_info, recursive, dvc_only)
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/repo/ls.py", line 69, in _ls
    metadata = fs.metadata(info)
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/fs/repo.py", line 409, in metadata
    raise FileNotFoundError
FileNotFoundError

I've tried updating to the latest dvc version, and tried this using a fresh clone of the git repo, but the error still exists.

Reproduce

The git repo is private and I can't really share the contents, but I basically do a git clone <repo>, and do a dvc list . -v.

Expected

It should list the files tracked by dvc.

Environment information

dvc==2.8.3

Output of dvc doctor:

$ dvc doctor

DVC version: 2.8.3 (pip)
---------------------------------
Platform: Python 3.8.11 on Linux-5.10.35-rt39-symbio-x86_64-with-glibc2.27
Supports:
	webhdfs (fsspec = 2021.11.1),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2021.6.1, boto3 = 1.17.49)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p5
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p5
Repo: dvc, git

Additional Information (if any):

The text was updated successfully, but these errors were encountered:

pared · 2021-12-13T18:46:26Z

@nikshar-symbio Hi!
I tried to analyze this use case but was unable to so far.
Would it be possible to provide the rest of the dvc list -v . log?

nikshar-symbio · 2021-12-13T18:52:36Z

Thanks @pared
Here's the full log

2021-12-13 10:51:50,512 ERROR: unexpected error
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/command/base.py", line 59, in do_run
    return self.run()
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/command/ls/__init__.py", line 30, in run
    entries = Repo.ls(
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/repo/ls.py", line 36, in ls
    ret = _ls(repo.repo_fs, path_info, recursive, dvc_only)
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/repo/ls.py", line 69, in _ls
    metadata = fs.metadata(info)
  File "/home/nikhil/.virtualenvs/init_py38/lib/python3.8/site-packages/dvc/fs/repo.py", line 484, in metadata
    raise FileNotFoundError
FileNotFoundError
------------------------------------------------------------
2021-12-13 10:51:50,692 DEBUG: Version info for developers:
DVC version: 2.8.3 (pip)
---------------------------------
Platform: Python 3.8.11 on Linux-5.10.35-rt39-symbio-x86_64-with-glibc2.27
Supports:
	webhdfs (fsspec = 2021.11.1),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2021.6.1, boto3 = 1.17.49)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p5
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p5
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-12-13 10:51:50,694 DEBUG: Analytics is enabled.
2021-12-13 10:51:50,757 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp3ufeozi3']'
2021-12-13 10:51:50,759 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp3ufeozi3']'

pared · 2021-12-13T19:07:17Z

Hmm, that does not help too much. @nikshar-symbio would you be open for a video call to debug it?
Feel free to ping me at pawel@iterative.ai

pared · 2021-12-15T10:32:44Z

Turns out its a loose symlink.

Reproduction script:

#!/bin/bash

set -ex
pushd $TMPDIR

w=test_wspace
rm -rf $w && mkdir $w && pushd $w

r=test_repo
mkdir $r && pushd $r

git init 
dvc init

touch file
ln -s file link

rm file
dvc list .

Related: iterative#7120

pared · 2021-12-17T12:32:07Z

#7157 is not a proper fix for this issue.
We should behave as ls would behave, so there should not be an error. Need to investigate why local fs thrown an error here.

Related: #7120

pared · 2022-01-04T10:09:10Z

related: fsspec/filesystem_spec#861

Fixes: iterative#7120

Fixes: #7120

karajan1001 · 2022-01-21T13:02:38Z

related to #5841?

george-kuanli-peng · 2023-06-07T07:30:00Z

I still have the same issue. I'm using dvc 2.58.2.

# dvc ls -v --dvc-only . ./.git
2023-06-07 07:14:39,308 DEBUG: v2.58.2 (pip), CPython 3.8.10 on Linux-5.4.0-84-generic-x86_64-with-glibc2.10
2023-06-07 07:14:39,309 DEBUG: command: /opt/conda/bin/dvc ls -v --dvc-only . ./.git
2023-06-07 07:14:40,565 ERROR: unexpected error
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/dvc/cli/__init__.py", line 210, in main
    ret = cmd.do_run()
  File "/opt/conda/lib/python3.8/site-packages/dvc/cli/command.py", line 40, in do_run
    return self.run()
  File "/opt/conda/lib/python3.8/site-packages/dvc/commands/ls/__init__.py", line 31, in run
    entries = Repo.ls(
  File "/opt/conda/lib/python3.8/site-packages/dvc/repo/ls.py", line 43, in ls
    ret = _ls(repo, path, recursive, dvc_only)
  File "/opt/conda/lib/python3.8/site-packages/dvc/repo/ls.py", line 62, in _ls
    fs_path = fs.info(fs_path)["name"]
  File "/opt/conda/lib/python3.8/site-packages/dvc_objects/fs/base.py", line 495, in info
    return self.fs.info(path, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/dvc/fs/dvc.py", line 330, in info
    return self._info(key, path, ignore_subrepos=ignore_subrepos)
  File "/opt/conda/lib/python3.8/site-packages/dvc/fs/dvc.py", line 371, in _info
    raise FileNotFoundError
FileNotFoundError

2023-06-07 07:14:40,826 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-06-07 07:14:40,826 DEBUG: Removing '/mlsteam/data/fake-dvc-folder/.bu325JV894Xgsa2Ur8oxVf.tmp'
2023-06-07 07:14:40,827 DEBUG: Removing '/mlsteam/data/fake-dvc-folder/.bu325JV894Xgsa2Ur8oxVf.tmp'
2023-06-07 07:14:40,827 DEBUG: Removing '/mlsteam/data/fake-dvc-folder/.bu325JV894Xgsa2Ur8oxVf.tmp'
2023-06-07 07:14:40,827 DEBUG: Removing '/mlsteam/data/fake-dvc-folder/VC-Blank-project/.dvc/cache/.db7GHdGMBsV3D2Vj6mkQvo.tmp'
2023-06-07 07:14:40,836 DEBUG: Version info for developers:
DVC version: 2.58.2 (pip)
-------------------------
Platform: Python 3.8.10 on Linux-5.4.0-84-generic-x86_64-with-glibc2.10
Subprojects:
        dvc_data = 0.51.0
        dvc_objects = 0.22.0
        dvc_render = 0.5.3
        dvc_task = 0.2.1
        scmrepo = 1.0.3
Supports:
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.5.0, boto3 = 1.26.76)
Config:
        Global: /mlsteam/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: xfs on /dev/sdb1
Caches: local
Remotes: s3
Workspace directory: xfs on /dev/sdb1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/481f8aaf689cfdbdb632a79cd882e7b2

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-06-07 07:14:40,839 DEBUG: Analytics is enabled.
2023-06-07 07:14:40,903 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp6s6ph8s0']'
2023-06-07 07:14:40,905 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp6s6ph8s0']'

dberenbaum · 2023-06-07T13:23:08Z

@george-kuanli-peng Do you mind opening in a new issue? It's easier for us to see it and prioritize it that way.

george-kuanli-peng · 2023-06-08T05:23:15Z

@george-kuanli-peng Do you mind opening in a new issue? It's easier for us to see it and prioritize it that way.

I opened a new issue #9564. Thanks for your suggestion.

pared added the bug Did we break something? label Dec 13, 2021

pared added the awaiting response we are waiting for your reply, please respond! :) label Dec 13, 2021

pared removed the awaiting response we are waiting for your reply, please respond! :) label Dec 14, 2021

pared mentioned this issue Dec 15, 2021

fs: repo: raise proper FileNotFoundError in metadata #7156

Merged

2 tasks

pared mentioned this issue Dec 15, 2021

exceptions: raise generic exceptions properly #7157

Closed

pared added a commit to pared/dvc that referenced this issue Dec 17, 2021

fs: repo: raise proper FileNotFoundError in metadata

f9469cb

Related: iterative#7120

pared self-assigned this Dec 17, 2021

efiop pushed a commit that referenced this issue Dec 17, 2021

fs: repo: raise proper FileNotFoundError in metadata

8ca3a7b

Related: #7120

pared added this to DVC Jan 4, 2022

pared moved this from Backlog to In Progress in DVC Jan 4, 2022

pared moved this to Backlog in DVC Jan 4, 2022

pared mentioned this issue Jan 5, 2022

ls: handle broken symlink #7226

Merged

2 tasks

pared added a commit to pared/dvc that referenced this issue Jan 6, 2022

ls: handle broken symlink

5f6755b

Fixes: iterative#7120

efiop closed this as completed in #7226 Jan 6, 2022

Repository owner moved this from In Progress to Done in DVC Jan 6, 2022

efiop pushed a commit that referenced this issue Jan 6, 2022

ls: handle broken symlink

fe4fefd

Fixes: #7120

daavoo added the A: status Related to the dvc diff/list/status label Feb 23, 2022

george-kuanli-peng mentioned this issue Jun 8, 2023

DVC list FileNotFoundError #9564

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DVC list FileNotFoundError #7120

DVC list FileNotFoundError #7120

nikshar-symbio commented Dec 9, 2021

pared commented Dec 13, 2021

nikshar-symbio commented Dec 13, 2021

pared commented Dec 13, 2021

pared commented Dec 15, 2021

pared commented Dec 17, 2021

pared commented Jan 4, 2022

karajan1001 commented Jan 21, 2022

george-kuanli-peng commented Jun 7, 2023

dberenbaum commented Jun 7, 2023

george-kuanli-peng commented Jun 8, 2023

DVC list FileNotFoundError #7120

DVC list FileNotFoundError #7120

Comments

nikshar-symbio commented Dec 9, 2021

Bug Report

dvc list . : gives "ERROR: unexpected error"

Description

Reproduce

Expected

Environment information

pared commented Dec 13, 2021

nikshar-symbio commented Dec 13, 2021

pared commented Dec 13, 2021

pared commented Dec 15, 2021

pared commented Dec 17, 2021

pared commented Jan 4, 2022

karajan1001 commented Jan 21, 2022

george-kuanli-peng commented Jun 7, 2023

dberenbaum commented Jun 7, 2023

george-kuanli-peng commented Jun 8, 2023

`dvc list . : gives "ERROR: unexpected error"`