PartialState().local_main_process_first() when map in examples

[qgallouedec]

No description provided.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Special attention here: do we need load_from_cache=False? I've removed them, otherwise it makes the loading in the main process first useless.

[lewtun]

Thanks for the improvement @qgallouedec !

If I recall correctly, we set load_from_cache_file=False to avoid a subtle side effect where a training script is run twice with a different number of samples in the dataset (e.g. imagine you train once on 50% of the data, followed by 100% of the data). What happens in that case is that datasets caches the results from map() in the first run and reuses them in the second one because the fingerprint hasn't changed (even though the number of samples has).

Having said that, I think it's an advanced use case and for sample scripts we can keep them simple as possible. (At some point I/we should see how to resolve the root cause on the datasets side)