fix: loading locally downloaded dataset

[NanoCode012]

Description

Fix #1830

Motivation and Context

How has this been tested?

Screenshots (if appropriate)

Types of changes

Social Handles (Optional)

[winglian]

If the previous test loading the dataset with snapshot download already worked, then didn't this already work? If not, the new test doesn't really seem to validate the new handling.

[NanoCode012]

If the previous test loading the dataset with snapshot download already worked, then didn't this already work? If not, the new test doesn't really seem to validate the new handling.

Which test do you mean in particular? This test_load_local_hub_with_revision?

If so, it's a bit different and also the reason I got confused with the original issue. The author wants to pass in the path to a local folder (without data_files) like so "path": str(tmp_ds_path),.

The prior test does

                            "path": "mhenrichsen/alpaca_2k_test",
                            "data_files": [
                                f"{tmp_ds_path}/alpaca_2000.parquet",
                            ],

They would enter separate conditions. Another way to test is that, if you revert the data loading changes, the new test would fail while rest passes.

[winglian]

thanks for the clarification 👍