gs: use 3-char prefixes

[isidentical]

Resolves #6691. Normally it should have inherited this from the ObjectFSWrapper, but this seems like a leftover.

[shcheklein]

Quick question/suggestion - is there a way/should we to add a test for this?

[isidentical]

Quick question/suggestion - is there a way/should we to add a test for this?

We already have 2 tests for this, but they cover the other way around.

dvc/tests/func/test_data_cloud.py

Lines 573 to 610 in dd86076

	@pytest.mark.parametrize("remote", full_clouds, indirect=True)
	def test_pull_00_prefix(tmp_dir, dvc, remote, monkeypatch):
	# Related: https://github.com/iterative/dvc/issues/6089
	fs_type = type(dvc.cloud.get_remote_odb("upstream").fs)
	monkeypatch.setattr(fs_type, "_ALWAYS_TRAVERSE", True, raising=False)
	monkeypatch.setattr(fs_type, "LIST_OBJECT_PAGE_SIZE", 256, raising=False)
	# foo's md5 checksum is 00411460f7c92d2124a67ea0f4cb5f85
	# bar's md5 checksum is 0000000018e6137ac2caab16074784a6
	tmp_dir.dvc_gen({"foo": "363", "bar": "jk8ssl"})
	dvc.push()
	clean(["foo", "bar"], dvc)
	stats = dvc.pull()
	assert stats["fetched"] == 2
	assert set(stats["added"]) == {"foo", "bar"}
	@pytest.mark.parametrize("remote", full_clouds, indirect=True)
	def test_pull_no_00_prefix(tmp_dir, dvc, remote, monkeypatch):
	# Related: https://github.com/iterative/dvc/issues/6244
	fs_type = type(dvc.cloud.get_remote_odb("upstream").fs)
	monkeypatch.setattr(fs_type, "_ALWAYS_TRAVERSE", True, raising=False)
	monkeypatch.setattr(fs_type, "LIST_OBJECT_PAGE_SIZE", 256, raising=False)
	# foo's md5 checksum is 14ffd92a6cbf5f2f657067df0d5881a6
	# bar's md5 checksum is 64020400f00960c0ef04052547b134b3
	tmp_dir.dvc_gen({"foo": "dvc", "bar": "cml"})
	dvc.push()
	clean(["foo", "bar"], dvc)
	stats = dvc.pull()
	assert stats["fetched"] == 2
	assert set(stats["added"]) == {"foo", "bar"}

[efiop]

Very interesting that this broke because of 2, as I would expect that to be less efficient but still work 🤔 @pmrowla do we have some logic flaw or am I missing something?

On a side note, once we migrate to fs.find I guess we could default to searching for 00/0* by default (non-obj-clouds will just search 00/ anyway) to simplify the logic.

[pmrowla]

Very interesting that this broke because of 2, as I would expect that to be less efficient but still work 🤔 @pmrowla do we have some logic flaw or am I missing something?

It seems like there is probably still some bug in the gcsfs prefix implementation somewhere (that doesn't like prefix len 2) rather than this being on DVC's end.

[efiop]

@pmrowla Was hoping to include a test here, strange that this hasn't been caught by our suit. Or do you think it should be tested on gcsfs side?

[pmrowla]

I think it needs to be tested on the gcsfs side. We use a mix of both 2 and 3 as prefix lengths on all of our other filesystems without seeing this issue.

[nikitaorlovpicsart]

after update start seening this issue on random files

[efiop]

@nikitaorlovpicsart , could you elaborate?

[nikitaorlovpicsart]

1. we use gs as remote
2. we use most recent release
3. issue mentioned in issue is still same for us, if we pull file directly, it works, but when we call pull in root folder, we get fail.

[nikitaorlovpicsart]

Every time all files start with 00

[efiop]

@nikitaorlovpicsart Could you show full verbose (-v) error you are getting? Also dvc doctor output, please.