dvc push: Unexpected error when pushing to Google Cloud storage or S3 #10206

turkanis · 2023-12-28T03:32:06Z

Bug Report

dvc push: "Unexpected error" when pushing to Google Cloud storage or S3

Reproduce

dvc init
dvc remote add -d s3 s3://bucket  # or gcs gs://bucket
dvc import-url https://data.dvc.org/get-started/data.xml
dvc push -v

output (s3):

2023-12-27 19:56:42,605 DEBUG: v3.36.1 (pip), CPython 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
2023-12-27 19:56:42,605 DEBUG: command: /path/bin/dvc push -v
Collecting                                                                                                                           |0.00 [00:00,    ?entry/s]
Pushing                                                                                                                              |0.00 [00:00,     ?file/s]
Collecting my.bucket/key on s3                                                                                                |3.00 [00:00, 4.84entry/s]
2023-12-27 19:56:43,676 ERROR: unexpected error
Traceback (most recent call last):
  File "/path/lib/python3.9/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/path/lib/python3.9/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/path/lib/python3.9/site-packages/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
  File "/path/lib/python3.9/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
  File "/path/lib/python3.9/site-packages/dvc/repo/push.py", line 144, in push
    push_transferred, push_failed = ipush(
  File "/path/lib/python3.9/site-packages/dvc_data/index/push.py", line 101, in push
    old = build(data.path, data.fs)
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 90, in build
    for entry in build_entries(path, fs, ignore=ignore):
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 55, in build_entries
    walk_iter = fs.walk(path, detail=detail)
  File "/path/lib/python3.9/site-packages/dvc_http/__init__.py", line 162, in walk
    raise NotImplementedError
NotImplementedError

2023-12-27 19:56:43,752 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-12-27 19:56:43,755 DEBUG: Removing '/path/.MHVNkr3eAijD7Q5aau3NRK.tmp'
2023-12-27 19:56:43,755 DEBUG: Removing '/path/.MHVNkr3eAijD7Q5aau3NRK.tmp'
2023-12-27 19:56:43,757 DEBUG: Removing '/path/.MHVNkr3eAijD7Q5aau3NRK.tmp'
2023-12-27 19:56:43,757 DEBUG: Removing '/path/bkw-9036/.dvc/cache/files/md5/.mnnSioPUuXvRUCqUV2ug87.tmp'
2023-12-27 19:56:43,777 DEBUG: Version info for developers:
DVC version: 3.36.1 (pip)
-------------------------
Platform: Python 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
Subprojects:
	dvc_data = 3.3.0
	dvc_objects = 3.0.0
	dvc_render = 1.0.0
	dvc_task = 0.3.0
	scmrepo = 2.0.2
Supports:
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.12.2, boto3 = 1.33.13)
Config:
	Global: /home/jdt/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/9d9135fb99d9d827364c4dc5a42cdc60

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-12-27 19:56:43,781 DEBUG: Analytics is enabled.
2023-12-27 19:56:43,860 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpccxiwrmd', '-v']
2023-12-27 19:56:43,871 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpccxiwrmd', '-v'] with pid 22406

output (gcs):

2023-12-27 19:47:22,768 DEBUG: v3.36.1 (pip), CPython 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
2023-12-27 19:47:22,769 DEBUG: command: /path/bin/dvc push -v
Collecting                                                                                                                           |0.00 [00:00,    ?entry/s]
Pushing                                                                                                                              |0.00 [00:00,     ?file/s]
Collecting bucket/path on gs                                                                                          |3.00 [00:01, 2.84entry/s]
2023-12-27 19:47:24,328 ERROR: unexpected error
Traceback (most recent call last):
  File "/path/lib/python3.9/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/path/lib/python3.9/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/path/lib/python3.9/site-packages/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
  File "/path/lib/python3.9/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
  File "/path/lib/python3.9/site-packages/dvc/repo/push.py", line 144, in push
    push_transferred, push_failed = ipush(
  File "/path/lib/python3.9/site-packages/dvc_data/index/push.py", line 101, in push
    old = build(data.path, data.fs)
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 90, in build
    for entry in build_entries(path, fs, ignore=ignore):
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 55, in build_entries
    walk_iter = fs.walk(path, detail=detail)
  File "/path/lib/python3.9/site-packages/dvc_http/__init__.py", line 162, in walk
    raise NotImplementedError
NotImplementedError

2023-12-27 19:47:24,370 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-12-27 19:47:24,371 DEBUG: Removing '/path/.fJ4uXqQznknWmbrzzUTXLQ.tmp'
2023-12-27 19:47:24,371 DEBUG: Removing '/path/.fJ4uXqQznknWmbrzzUTXLQ.tmp'
2023-12-27 19:47:24,371 DEBUG: Removing '/path/.fJ4uXqQznknWmbrzzUTXLQ.tmp'
2023-12-27 19:47:24,371 DEBUG: Removing '/path/bkw-9036/.dvc/cache/files/md5/.M6iwnJkjQgKzg54kN6chVi.tmp'
2023-12-27 19:47:24,377 DEBUG: Version info for developers:
DVC version: 3.36.1 (pip)
-------------------------
Platform: Python 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
Subprojects:
	dvc_data = 3.3.0
	dvc_objects = 3.0.0
	dvc_render = 1.0.0
	dvc_task = 0.3.0
	scmrepo = 2.0.2
Supports:
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3)
Config:
	Global: /home/jdt/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: gs
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/9d9135fb99d9d827364c4dc5a42cdc60

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-12-27 19:47:24,379 DEBUG: Analytics is enabled.
2023-12-27 19:47:24,445 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpk_30nnlt', '-v']
2023-12-27 19:47:24,455 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpk_30nnlt', '-v'] with pid 15755

Expected

Successful push

Environment information

DVC version: 3.36.1 (pip)
-------------------------
Platform: Python 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
Subprojects:
	dvc_data = 3.3.0
	dvc_objects = 3.0.0
	dvc_render = 1.0.0
	dvc_task = 0.3.0
	scmrepo = 2.0.2
Supports:
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.12.2, boto3 = 1.33.13)
Config:
	Global: /home/jdt/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/c9c73dbc105eb09a15137f49a60e6a5b

Additional Information (if any):

The text was updated successfully, but these errors were encountered:

efiop · 2023-12-28T04:14:01Z

Looks like we are accidentally trying to push to that import-url's fs. Taking a look...

efiop · 2023-12-28T04:27:04Z

Ok, with index migration we need storage.read_only that we can use during index building to say that this remote is read-only. This is similar to odb-only read_only that we used to have in the past. Already preparing a fix.

skshetry · 2023-12-28T04:30:04Z

Is it about read-only? Or, is it about traversable/non-traversable remotes? Our implementation of http is read+write.

efiop · 2023-12-28T04:38:08Z

@skshetry We are trying to push to http that we've import-url-ed from and which is not even set up as a remote, so the former.

Fixes iterative#10206

Fixes #10206

efiop · 2023-12-28T06:09:18Z

@turkanis Could you try upstream dvc, please? https://dvc.org/doc/install/pre-release Should be fixed now.

turkanis · 2023-12-28T23:22:38Z

@efiop

Thanks for your quick response.

With the pre-release version, I no longer get an error when I run dvc push. However, no push to remote storage occurs. Instead, I get this output:

Collecting                                        |0.00 [00:00,    ?entry/s]
Pushing
Everything is up to date.

Is this the expected behavior? It is not what I expected. If, instead, I add the --to-remote option to dvc url-import, the filve is added to remote storage, and I can download it with dvc pull.

To summarize, without --to-remote, the file is copied to the project directory, a .dvc tracking file is created, but the data never gets written to remote storage. Running dvc import-url with --to-remote followed by dvc pull gives me the behavior I was expecting initially from dvc import-url

efiop · 2023-12-29T03:28:42Z

@turkanis If you've pushed already and there are no new files to push then it is expected that it is not pushing anything.

--to-remote works in a different way and will indeed do something every time, because it is assuming that you are sure that that file is not present in the remote yet.

turkanis · 2023-12-29T03:32:36Z

@efiop

In the scenario I described, I had never pushed before, and there were no objects in remote storage.

skshetry · 2023-12-29T03:43:09Z

@turkanis, you might be facing #10194.

turkanis · 2023-12-29T16:32:49Z

@efiop

I think I have not described the situation well. Here are the 2 scenarios that occur. In both cases, some of the behavior seems wrong to me. Both start out with nothing in remote storage:

Scenario I

 git init
 dvc init
 dvc remote add -d gcs gs://bucket/dvc-test
 dvc import-url https://data.dvc.org/get-started/data.xml

   # creates data.xml
   # creates data.xml.dvc
   # adds entry in .gitignore

 dvc push 

   # nothing written to remote storage

Scenario II

git init
dvc init
dvc remote add -d gcs gs://bucket/dvc-test
dvc import-url --to-remote https://data.dvc.org/get-started/data.xml

	# creates data.xml.dvc
	# no entry added to .gitignore
	
dvc pull

	# creates data.xml
	# git shows data.xml as untracked

Is this expected behavior? What I want is the imported file to be written to remote storage and to have a .gitignore entry created automatically.

Thanks for your help!

efiop · 2023-12-30T01:23:11Z

@turkanis Thanks for a detailed description! First one will be solved in #10194 (I left a comment there) and the second one looks like a separate bug. Could you create an issue for the latter one, please?

turkanis · 2024-01-03T01:17:31Z

@efiop

I have created #10217

Fixes iterative#10206

efiop self-assigned this Dec 28, 2023

efiop added the bug Did we break something? label Dec 28, 2023

efiop added this to DVC Dec 28, 2023

github-project-automation bot moved this to Backlog in DVC Dec 28, 2023

efiop added the p0-critical Critical issue. Needs to be fixed ASAP. label Dec 28, 2023

efiop mentioned this issue Dec 28, 2023

index: introduce Storage.read_only iterative/dvc-data#491

Merged

efiop added a commit to efiop/dvc that referenced this issue Dec 28, 2023

index: mark remote as read_only for imports

7cde090

Fixes iterative#10206

efiop mentioned this issue Dec 28, 2023

index: mark remote as read_only for imports #10208

Merged

efiop added a commit to efiop/dvc that referenced this issue Dec 28, 2023

index: mark remote as read_only for imports

4cf131b

Fixes iterative#10206

efiop closed this as completed in #10208 Dec 28, 2023

efiop added a commit that referenced this issue Dec 28, 2023

index: mark remote as read_only for imports

f2e4794

Fixes #10206

github-project-automation bot moved this from Backlog to Done in DVC Dec 28, 2023

BradyJ27 pushed a commit to BradyJ27/dvc that referenced this issue Apr 22, 2024

index: mark remote as read_only for imports

ce19b63

Fixes iterative#10206

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dvc push: Unexpected error when pushing to Google Cloud storage or S3 #10206

dvc push: Unexpected error when pushing to Google Cloud storage or S3 #10206

turkanis commented Dec 28, 2023 •

edited by efiop

Loading

efiop commented Dec 28, 2023

efiop commented Dec 28, 2023

skshetry commented Dec 28, 2023

efiop commented Dec 28, 2023

efiop commented Dec 28, 2023

turkanis commented Dec 28, 2023

efiop commented Dec 29, 2023

turkanis commented Dec 29, 2023

skshetry commented Dec 29, 2023

turkanis commented Dec 29, 2023

efiop commented Dec 30, 2023

turkanis commented Jan 3, 2024

dvc push: Unexpected error when pushing to Google Cloud storage or S3 #10206

dvc push: Unexpected error when pushing to Google Cloud storage or S3 #10206

Comments

turkanis commented Dec 28, 2023 • edited by efiop Loading

Bug Report

Reproduce

Expected

Environment information

efiop commented Dec 28, 2023

efiop commented Dec 28, 2023

skshetry commented Dec 28, 2023

efiop commented Dec 28, 2023

efiop commented Dec 28, 2023

turkanis commented Dec 28, 2023

efiop commented Dec 29, 2023

turkanis commented Dec 29, 2023

skshetry commented Dec 29, 2023

turkanis commented Dec 29, 2023

Scenario I

Scenario II

efiop commented Dec 30, 2023

turkanis commented Jan 3, 2024

turkanis commented Dec 28, 2023 •

edited by efiop

Loading