FolderBase Dataset automatically resolves under current directory when data_dir is not specified

[npuichigo]

Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:

load_dataset("audiofolder")

takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling

datasets/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py

Lines 58 to 59 in cb8c5de

	if not self.config.data_files:
	raise ValueError(f"At least one data file must be specified, but got data_files={self.config.data_files}")

Steps to reproduce the bug

load_dataset("audiofolder")

Expected behavior

Error report

Environment info

• datasets version: 2.14.4
• Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
• Python version: 3.8.15
• Huggingface_hub version: 0.16.4
• PyArrow version: 12.0.1
• Pandas version: 1.5.3

[npuichigo]

@lhoestq

[lhoestq]

Makes sense, I guess this can be fixed in the load_dataset_builder method.
It concerns every packaged builder I think (see values in _PACKAGED_DATASETS_MODULES)

[npuichigo]

I think the behavior is related to these lines, which short circuited the error handling.

datasets/src/datasets/load.py

Lines 946 to 952 in 664a1cb

	base_path = Path(self.data_dir or "").expanduser().resolve().as_posix()
	patterns = sanitize_patterns(self.data_files) if self.data_files is not None else get_data_patterns(base_path)
	data_files = DataFilesDict.from_patterns(
	patterns,
	download_config=self.download_config,
	base_path=base_path,
	)

So should data_dir be checked here or still delegating to actual DatasetModule? In that case, how to properly set data_files here.

[lhoestq]

This is location in PackagedDatasetModuleFactory.get_module seems the be the right place to check if at least data_dir or data_files are passed

[debrupf2946]

@mariosasko can you please assign this issue to me,I want to work on this

[debrupf2946]

#self-assign

[zutarich]

@mariosasko is this issue still open? i would love to kickstart my journey to open source with this issue!
Regards
zutarich

[mariosasko]

@zutarich It is unless @debrupf2946 is working on it.

[Etelis]

#self-assign

[debrupf2946]

I am working and will open a pull request soon @Etelis

[JINO-ROHIT]

@mariosasko can i take this up?

[JINO-ROHIT]

#self-assign

[mariosasko]

Yes, feel free to work on this :)

[JINO-ROHIT]

i think its working as expected . Heres the log i get for the same line -

[sahillihas]

#self-assign