Rank and files

[tobi]

This does two things:

• Allow dataset_format to be specified as input-output. I was also considering to call it custom. But basically it's just "don't do anything". I feel like this is the correct default in cases where a local dataset is being passed in because it's likely that the file can be formatted however one wishes and the docs say to make it input/output which is a sensible default
• We also look at LOCAL_RANK to set the device_map to force everything to a single device. This allows torch run to do it's work. (admittedly, I'm a noob on this stuff, but it does seem to work well enough)

[lhoestq]

This change would load the full dataset in memory, is it intentional ?

[artidoro]

Typically, NLP datasets are small enough to fit in memory, so this should be fine in most cases. However, I am unaware of the benefits of using Pandas vs HF Datasets for loading and have not benchmarked the two libraries. Could you provide some more details? Otherwise, I lean towards using the HF Datasets method.

[tobi]

I think using datasets library directly would be better, the previous code didn't work though so I fixed it the way I knew how to. You are right that it would be better to just correct the syntax.

[lhoestq]

HF Datasets converts the data to an Arrow file and memory maps the data from disk. This gives high speed while keeping the RAM usage to minimum. It's also useful in distributed setups because the memory mapped file can be seen as shared memory across processes - no need to copy the data to the different processes.

[tobi]

I'd switch it but I'm traveling rest of week.

[artidoro]

Thank you @tobi! I agree the default format should be input-output and the device map specification looks good to me. It will be useful to have it working with torch run!

[artidoro]

Thank you for your contributions!