dvc pull returns "failed to pull data" when the data exists on remote

[RadionBik]

Bug Report

Issue name

dvc pull returns "failed to pull data" when the data exists on remote

Description

dvc pull (also tried with -R option) fails to pull remote data basing on .dvc files from sub-directories and returns ERROR: failed to pull data from the cloud - Checkout failed for following targets:..., however, when I run the pull cmd on failed files individually, the cmd succeeds.

(onboarding_models) radion@MacBook-Pro-Radion anna-datascience % dvc pull
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:                                                                                                                                                                                                                                                                    
name: document_labelling_utils/annotation_results/1000_recent_documents_20210413.json, md5: 06a0a6ef5b6446a33623a544ede8bbfd
1 file failed                                                                                                                                                                                                                                                                                                                                                        
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
document_labelling_utils/annotation_results/1000_recent_documents_20210413.json
Is your cache up to date?
<https://error.dvc.org/missing-files>
(onboarding_models) radion@MacBook-Pro-Radion anna-datascience % dvc pull -R
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:                                                                                                                                                                                                                                                                    
name: document_labelling_utils/annotation_results/1000_recent_documents_20210413.json, md5: 06a0a6ef5b6446a33623a544ede8bbfd
1 file failed                                                                                                                                                                                                                                                                                                                                                        
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
document_labelling_utils/annotation_results/1000_recent_documents_20210413.json
Is your cache up to date?
<https://error.dvc.org/missing-files>
(onboarding_models) radion@MacBook-Pro-Radion anna-datascience % dvc pull document_labelling_utils/annotation_results/1000_recent_documents_20210413.json.dvc 
A       document_labelling_utils/annotation_results/1000_recent_documents_20210413.json                                                                                                                                                                                                                                                                              
1 file added and 1 file fetched                                                                                                                                                                                                                                                                                                                                      
(onboarding_models) radion@MacBook-Pro-Radion anna-datascience % dvc pull                                                                                    
Everything is up to date.                                                                                                                                                                                                                                                                                                                                            

Expected

I expect dvc pull to download missing files from sub-directories without the need to run it on each .dvc file.

Environment information

Output of dvc doctor:

DVC version: 2.7.2 (brew)
---------------------------------
Platform: Python 3.9.7 on macOS-11.2.1-x86_64-i386-64bit
Supports:
        azure (adlfs = 2021.8.2, knack = 0.8.2, azure-identity = 1.6.1),
        gdrive (pydrive2 = 1.9.3),
        gs (gcsfs = 2021.8.1),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        s3 (s3fs = 2021.8.1, boto3 = 1.17.106),
        webdav (webdav4 = 0.9.1),
        webdavs (webdav4 = 0.9.1)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: gs
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git

[pmrowla]

@RadionBik are your .dvc files inside .gitignore'd directories?

[RadionBik]

no, the directories are not git-ignored.

[clementperon]

Hi,

I think I have the same issue.

Pulling with the cmd
dvc pull -f data/toto.dvc data/tutu.dvc data/tete.dvc

Will create a warning
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:

If I pull each file one by one it's OK.

$> dvc doctor

DVC version: 2.7.4 (pip)
---------------------------------
Platform: Python 3.9.5 on Linux-5.13.0-7614-generic-x86_64-with-glibc2.33
Supports:
	gs (gcsfs = 2021.8.1),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5)

[efiop]

@clementperon Could you show the contents of those dvc files, please? Also, full verbose error log would be really helpful.

[clementperon]

$> cat data/2021_06_29_some_text_here_in_snakecase.pcap.dvc

- md5: 47a9b6d2693147f689ff5ebe12c78a05
  size: 2284260388
  path: 2021_06_29_some_text_here_in_snakecase.pcap

[clementperon]

$ dvc pull -f data/file_1.pcap.dvc data/file_2.ply.dvc
Everything is up to date.
$ rm .dvc/cache/47/a9b6d2693147f689ff5ebe12c78a05
rm: remove write-protected regular file '.dvc/cache/47/a9b6d2693147f689ff5ebe12c78a05'? y
$ dvc pull -f data/file_1.pcap.dvc data/file_2.ply.dvc

WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:                                                                                                                                                             
name: data/file_1.pcap, md5: 47a9b6d2693147f689ff5ebe12c78a05
ERROR: unexpected error                                                                                                                                                                                                                                       

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

$ dvc pull -f data/file_1.pcap.dvc

  0% Transferring| 
|0/1 [00:00<?,     ?file/s^C14%|█▎        |a9b6d2693147f689ff5ebe12c78a05

[clementperon]

Please let me know, if you want more details / cmd to execute.

[pmrowla]

I'm unable to reproduce this with local or s3 remotes, it may be something specific to gs/gcsfs

[clementperon]

The issue has been introduced in 2.5.0.

$> pip install dvc[gs]==2.4.3
dvc pull -f toto1.dvc toto2.dvc

Is OK

$> pip install dvc[gs]==2.5.0
dvc pull -f toto1.dvc toto2.dvc

Failed !!

[clementperon]

Doing git bisect:

Turn this commit trig the issue
commit e3ff6d5 (refs/bisect/bad)
Author: Batuhan Taskaya batuhanosmantaskaya@gmail.com
Date: Thu Jul 1 14:59:35 2021 +0300

fsspec: loosen the prefix check to cover both None and False (#6246)

* fsspec: loosen the prefix check to cover both None and False

* gs: disable prefix-based search

Can confirm:

$> git reset --hard 2.5.0
$> git revert e3ff6d5
$> python3 -m build
$> python3 -m pip install dist/dvc-2.5.0+7d239d.tar.gz
$> dvc pull -f data1.dvc data2.dvc

Works fine !

[pmrowla]

cc @isidentical

[clementperon]

Looks like removing the TRAVERSE_PREFIX_LEN fix my issue.

diff --git a/dvc/fs/gs.py b/dvc/fs/gs.py
index 6ee6d735..2d73f063 100644
--- a/dvc/fs/gs.py
+++ b/dvc/fs/gs.py
@@ -16,7 +16,6 @@ class GSFileSystem(CallbackMixin, ObjectFSWrapper):
     REQUIRES = {"gcsfs": "gcsfs"}
     PARAM_CHECKSUM = "etag"
     DETAIL_FIELDS = frozenset(("etag", "size"))
-    TRAVERSE_PREFIX_LEN = 2
 
     def _prepare_credentials(self, **config):
         login_info = {"consistency": None}

Tested on master.

[efiop]

@clementperon Thank you for the research! 🙏

[clementperon]

@efiop Pleasure is mine 🙂.