fsspec: support fsspec>=2023.12.0 glob changes #6687

pmrowla · 2024-02-22T08:59:32Z

adds support for the fs.glob changes introduced in fsspec==2023.12.0 and unpins the current upper bound

Should close #6644
Should close #6645

The test_data_files glob/pattern tests pass for me in:

fsspec==2023.10.0 (the pinned max version in datasets main)
fsspec==2023.12.0 (Support fsspec 2023.12 #6644)
fsspec==2024.2.0 (Support fsspec 2024.2 #6645)

pmrowla · 2024-02-22T09:01:39Z

setup.py

@@ -131,7 +131,7 @@
    "multiprocess",
    # to save datasets locally or on any filesystem
    # minimum 2023.1.0 to support protocol=kwargs in fsspec's `open`, `get_fs_token_paths`, etc.: see https://github.com/fsspec/filesystem_spec/pull/1143
-    "fsspec[http]>=2023.1.0,<=2023.10.0",
+    "fsspec[http]>=2023.1.0",


Was unsure if the preference here is to remove the upper bound or to increase the range to the current fsspec release (<=2024.2.0)

We prefer using <=2024.2.0, since a breaking change in fsspec can happen in any monthly version

HuggingFaceDocBuilderDev · 2024-02-22T16:13:20Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lhoestq · 2024-02-22T16:22:11Z

src/datasets/data_files.py

@@ -134,6 +144,23 @@ def sanitize_patterns(patterns: Union[Dict, List, str]) -> Dict[str, Union[List[
        return sanitize_patterns(list(patterns))


+def _expand_globstar(pattern: str) -> str:


I think we should fully align with the new fsspec logic, so ideally it would be great to remove the extra _expand_globstar logic.

Therefore it's fine to simply remove the test examples using unsupported patterns like data/**, **train* or **test*, and and update the remaining patterns that still use ** like DEFAULT_PATTERNS_ALL and METADATA_PATTERNS (and some patterns from the new KEYWORDS_IN_PATH_NAME_BASE_PATTERNS you added) EDIT: actually only remove **train* and **test* in the tests

What do you think ?

This is fine with me, I was just going off of the original comment from #6644 which said the intention was to continue supporting the old glob behavior

cc @albertvillanova IMO we can drop the <prefix>**<suffix> support, since it's been a behavior that only existed in fsspec and no other well known globbing tool afaik

I double checked and actually we just need to remove the test involving **train* and **test*.

No need to modify data/**, DEFAULT_PATTERNS_ALL, METADATA_PATTERNS or KEYWORDS_IN_PATH_NAME_BASE_PATTERNS as I said in my previous message.

I think KEYWORDS_IN_PATH_NAME_BASE_PATTERNS does still need to be modified, I still see test failures with the existing base patterns

FAILED tests/test_data_files.py::test_get_data_files_patterns[data_file_per_split0-] - ValueError: Invalid pattern: '**' can only be an entire path component

It looks to me like {keyword}[{sep}/]** still generates {keyword}** which is invalid in fsspec>=2023.12.0

@lhoestq I've pushed an update that removes the unsupported **train*/**test* tests, removes the _expand_globstar behavior, and simplifies the updated keyword patterns to

KEYWORDS_IN_PATH_NAME_BASE_PATTERNS = [ "**/{keyword}[{sep}]*", "**/{keyword}/**", "**/*[{sep}]{keyword}[{sep}]*", "**/*[{sep}]{keyword}[{sep}]*/**", ]

pmrowla · 2024-02-28T05:59:47Z

Looking into the CI failure, this PR is incompatible with huggingface-hub>=0.20.0. It looks like there were several changes made to HfFileSystem in 0.20.0, @lhoestq any ideas on what the issue might be in particular?

a bisect indicates that it's related to huggingface/huggingface_hub#1815

pmrowla · 2024-02-28T06:33:00Z

It looks like huggingface-hub's HfFileSystem.glob is broken for exact string matches (that don't contain glob wildcards) when combining huggingface-hub>=0.20.0 and fsspec>=2023.12.0.

I did a quick test with huggingface-hub main, and adding this test case to tests/test_hf_filesystem::HfFileSystemTests::test_glob (https://github.com/huggingface/huggingface_hub/blob/main/tests/test_hf_file_system.py) passes with fsspec==2023.10.0 and fails with fsspec==2023.12.0

        self.assertEqual(
            sorted(self.hffs.glob(self.hf_path + "/.gitattributes")),
            sorted([self.hf_path + "/.gitattributes"]),
        )

the hffs.glob() call with a pattern that does not contain any wildcards returns an empty list:

E       AssertionError: Lists differ: [] != ['datasets/__DUMMY_TRANSFORMERS_USER__/rep[35 chars]tes']
E
E       Second list contains 1 additional elements.
E       First extra element 0:
E       'datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes'
E
E       - []
E       + ['datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes']

(and with the compatible/passing older fsspec versions the glob call returns the single exact file match as expected)

So it looks like the CI failure here isn't directly related to this PR. The failing patterns that don't contain any * wildcards are generated by datasets with or without this PR, but now that this PR installs the incompatible fsspec version, the underlying HfFileSystem.glob() call ends up failing.

lhoestq · 2024-02-28T15:08:54Z

I just opened huggingface/huggingface_hub#2056 to fix this.

Do you mind if I continue this PR to run the CI against huggingface_hub@main until the fix is released ?

EDIT: the fix has been released in huggingface_hub 0.21.2 - I removed my commits that were using huggingface_hub@main

lhoestq · 2024-02-29T10:57:05Z

I just added two additional patterns to cover cases like test-data/xxx.csv and data-test/xxx.csv

lhoestq

Alright it looks all good now, thanks for the fix ! also cc @mariosasko

github-actions · 2024-02-29T15:18:31Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005461 / 0.011353 (-0.005892)	0.003861 / 0.011008 (-0.007148)	0.063252 / 0.038508 (0.024744)	0.031474 / 0.023109 (0.008364)	0.250321 / 0.275898 (-0.025577)	0.275198 / 0.323480 (-0.048282)	0.003275 / 0.007986 (-0.004710)	0.002874 / 0.004328 (-0.001454)	0.049499 / 0.004250 (0.045248)	0.045334 / 0.037052 (0.008282)	0.266347 / 0.258489 (0.007858)	0.308974 / 0.293841 (0.015133)	0.027742 / 0.128546 (-0.100804)	0.010274 / 0.075646 (-0.065373)	0.207516 / 0.419271 (-0.211755)	0.036538 / 0.043533 (-0.006995)	0.247949 / 0.255139 (-0.007190)	0.268986 / 0.283200 (-0.014214)	0.019842 / 0.141683 (-0.121841)	1.117547 / 1.452155 (-0.334607)	1.175813 / 1.492716 (-0.316903)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.103661 / 0.018006 (0.085655)	0.331023 / 0.000490 (0.330534)	0.000240 / 0.000200 (0.000040)	0.000041 / 0.000054 (-0.000013)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019767 / 0.037411 (-0.017645)	0.061500 / 0.014526 (0.046974)	0.075899 / 0.176557 (-0.100658)	0.122240 / 0.737135 (-0.614895)	0.074621 / 0.296338 (-0.221717)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.287501 / 0.215209 (0.072292)	2.794737 / 2.077655 (0.717082)	1.505362 / 1.504120 (0.001242)	1.379481 / 1.541195 (-0.161713)	1.394836 / 1.468490 (-0.073654)	0.545803 / 4.584777 (-4.038974)	2.364167 / 3.745712 (-1.381545)	2.800923 / 5.269862 (-2.468939)	1.723910 / 4.565676 (-2.841766)	0.061270 / 0.424275 (-0.363005)	0.005006 / 0.007607 (-0.002601)	0.334952 / 0.226044 (0.108908)	3.367122 / 2.268929 (1.098194)	1.839822 / 55.444624 (-53.604803)	1.553774 / 6.876477 (-5.322703)	1.583585 / 2.142072 (-0.558488)	0.624680 / 4.805227 (-4.180547)	0.116364 / 6.500664 (-6.384300)	0.042412 / 0.075469 (-0.033057)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.975207 / 1.841788 (-0.866580)	11.843126 / 8.074308 (3.768818)	9.418537 / 10.191392 (-0.772855)	0.130648 / 0.680424 (-0.549775)	0.013747 / 0.534201 (-0.520454)	0.288195 / 0.579283 (-0.291088)	0.269861 / 0.434364 (-0.164503)	0.326732 / 0.540337 (-0.213606)	0.441256 / 1.386936 (-0.945680)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005185 / 0.011353 (-0.006168)	0.003836 / 0.011008 (-0.007172)	0.050057 / 0.038508 (0.011549)	0.030929 / 0.023109 (0.007820)	0.263558 / 0.275898 (-0.012340)	0.284553 / 0.323480 (-0.038927)	0.004331 / 0.007986 (-0.003655)	0.002815 / 0.004328 (-0.001513)	0.050187 / 0.004250 (0.045936)	0.048431 / 0.037052 (0.011379)	0.271005 / 0.258489 (0.012515)	0.304749 / 0.293841 (0.010908)	0.029286 / 0.128546 (-0.099260)	0.010598 / 0.075646 (-0.065048)	0.058111 / 0.419271 (-0.361160)	0.053665 / 0.043533 (0.010132)	0.257574 / 0.255139 (0.002436)	0.285802 / 0.283200 (0.002602)	0.018917 / 0.141683 (-0.122766)	1.206517 / 1.452155 (-0.245638)	1.220572 / 1.492716 (-0.272144)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.122466 / 0.018006 (0.104460)	0.567887 / 0.000490 (0.567397)	0.000321 / 0.000200 (0.000121)	0.000049 / 0.000054 (-0.000006)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022120 / 0.037411 (-0.015292)	0.075456 / 0.014526 (0.060931)	0.086385 / 0.176557 (-0.090171)	0.126045 / 0.737135 (-0.611091)	0.087502 / 0.296338 (-0.208837)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.304847 / 0.215209 (0.089638)	3.008095 / 2.077655 (0.930441)	1.726178 / 1.504120 (0.222058)	1.592332 / 1.541195 (0.051138)	1.603714 / 1.468490 (0.135224)	0.576875 / 4.584777 (-4.007902)	2.450884 / 3.745712 (-1.294828)	2.719073 / 5.269862 (-2.550789)	1.775261 / 4.565676 (-2.790415)	0.063144 / 0.424275 (-0.361131)	0.005122 / 0.007607 (-0.002485)	0.350004 / 0.226044 (0.123960)	3.467146 / 2.268929 (1.198218)	2.062907 / 55.444624 (-53.381717)	1.798793 / 6.876477 (-5.077684)	1.921204 / 2.142072 (-0.220868)	0.651832 / 4.805227 (-4.153396)	0.122326 / 6.500664 (-6.378338)	0.041396 / 0.075469 (-0.034073)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.024859 / 1.841788 (-0.816928)	12.569744 / 8.074308 (4.495436)	10.448487 / 10.191392 (0.257095)	0.131529 / 0.680424 (-0.548895)	0.014853 / 0.534201 (-0.519348)	0.287683 / 0.579283 (-0.291600)	0.289814 / 0.434364 (-0.144550)	0.323935 / 0.540337 (-0.216403)	0.425208 / 1.386936 (-0.961728)

albertvillanova · 2024-03-04T12:38:46Z

EDIT: the fix has been released in huggingface_hub 0.21.2 - I removed my commits that were using huggingface_hub@main

Please note that people using huggingface_hub < 0.21.2 and latest fsspec will have issues when using datasets:

CC: @clefourrier

pmrowla added 2 commits February 22, 2024 18:00

data_files: support fsspec 2023.12.0 glob

48ac4e1

fsspec: unpin version upper bound

ceb7a04

pmrowla force-pushed the fsspec-2023.12.0-glob branch from c79dba2 to ceb7a04 Compare February 22, 2024 09:00

pmrowla commented Feb 22, 2024

View reviewed changes

pmrowla marked this pull request as ready for review February 22, 2024 09:05

lhoestq reviewed Feb 22, 2024

View reviewed changes

skshetry mentioned this pull request Feb 23, 2024

remote: issue-8642 fix. DVC pull from HDFS error (4.0.1->4.0.2) iterative/dvc-objects#281

Closed

pmrowla added 3 commits February 26, 2024 14:46

fsspec: pin max version to <=2024.2.0

a28ca6e

data_files: remove unsupported fsspec-specific ** globbing

99992e7

data_files: update resolve_pattern ** behavior docstring

c5b4803

lhoestq mentioned this pull request Feb 28, 2024

[HfFileSystem] Fix glob with pattern without wildcards huggingface/huggingface_hub#2056

Merged

lhoestq force-pushed the fsspec-2023.12.0-glob branch 2 times, most recently from f8059cf to c5b4803 Compare February 28, 2024 15:53

fix split case with either prefix or suffix

434e026

lhoestq approved these changes Feb 29, 2024

View reviewed changes

Merge branch 'main' into fsspec-2023.12.0-glob

18816cc

lhoestq merged commit ba71e92 into huggingface:main Feb 29, 2024
2 checks passed

pmrowla deleted the fsspec-2023.12.0-glob branch March 4, 2024 12:59

albertvillanova mentioned this pull request Mar 4, 2024

Bump huggingface-hub lower version to 0.21.2 #6713

Merged

albertvillanova mentioned this pull request May 13, 2024

Invalid pattern: '**' can only be an entire path component #6737

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fsspec: support fsspec>=2023.12.0 glob changes #6687

fsspec: support fsspec>=2023.12.0 glob changes #6687

pmrowla commented Feb 22, 2024 •

edited

Loading

pmrowla Feb 22, 2024

lhoestq Feb 22, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Feb 22, 2024

lhoestq Feb 22, 2024 •

edited

Loading

pmrowla Feb 23, 2024

lhoestq Feb 23, 2024

pmrowla Feb 26, 2024

pmrowla Feb 26, 2024

pmrowla commented Feb 28, 2024 •

edited

Loading

pmrowla commented Feb 28, 2024 •

edited

Loading

lhoestq commented Feb 28, 2024 •

edited

Loading

lhoestq commented Feb 29, 2024

lhoestq left a comment

github-actions bot commented Feb 29, 2024

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

albertvillanova commented Mar 4, 2024

		@@ -134,6 +144,23 @@ def sanitize_patterns(patterns: Union[Dict, List, str]) -> Dict[str, Union[List[
		return sanitize_patterns(list(patterns))


		def _expand_globstar(pattern: str) -> str:

fsspec: support fsspec>=2023.12.0 glob changes #6687

fsspec: support fsspec>=2023.12.0 glob changes #6687

Conversation

pmrowla commented Feb 22, 2024 • edited Loading

pmrowla Feb 22, 2024

Choose a reason for hiding this comment

lhoestq Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Feb 22, 2024

lhoestq Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

pmrowla Feb 23, 2024

Choose a reason for hiding this comment

lhoestq Feb 23, 2024

Choose a reason for hiding this comment

pmrowla Feb 26, 2024

Choose a reason for hiding this comment

pmrowla Feb 26, 2024

Choose a reason for hiding this comment

pmrowla commented Feb 28, 2024 • edited Loading

pmrowla commented Feb 28, 2024 • edited Loading

lhoestq commented Feb 28, 2024 • edited Loading

lhoestq commented Feb 29, 2024

lhoestq left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 29, 2024

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

albertvillanova commented Mar 4, 2024

pmrowla commented Feb 22, 2024 •

edited

Loading

lhoestq Feb 22, 2024 •

edited

Loading

lhoestq Feb 22, 2024 •

edited

Loading

pmrowla commented Feb 28, 2024 •

edited

Loading

pmrowla commented Feb 28, 2024 •

edited

Loading

lhoestq commented Feb 28, 2024 •

edited

Loading