Support custom split name by renaming files

[wengh]

Overview

This PR adds support for custom split names by renaming files to follow the file naming convention of HuggingFace datasets.

What's changed

In append mode we now make an additional commit to rename existing files of the split to have the correct name, e.g. with updated total number of files.

Then we commit the newly uploaded files directly with the correct name, and (in overwrite mode) delete old files that are not overwritten.

Append requires an additional commit because in a single commit we cannot rename a file A->B while creating a new file with name A.

Walkthrough (append mode)

Suppose that the repo initially contains

- data/
    - custom-00000-of-00002.parquet
    - custom-00001-of-00002.parquet

We use the data source to append a dataframe of 1 partition to the split:

>>> df.write.format("huggingfacesink").option(token=..., split="custom").save(...)

HuggingFaceDatasetsWriter.write first pre-uploads the new parquet file under a unique temporary name. This uploads the file to LFS but doesn't commit it to the repo.

After all partitions have been pre-uploaded, HuggingFaceDatasetsWriter.commit gets called. It first lists all existing files of the split, renaming them to the the correct format (changing total count from 00002 to 00003):

- data/
    - custom-00000-of-00003.parquet (moved from custom-00000-of-00002.parquet)
    - custom-00001-of-00003.parquet (moved from custom-00001-of-00002.parquet)

Then, it uploads the new file under the correct name, in a separate commit:

- data/
    - custom-00000-of-00003.parquet
    - custom-00001-of-00003.parquet
    - custom-00002-of-00003.parquet (new)

Testing

This PR updates tests to test for custom split names. It also replaces reader with direct call to datasets for slightly less overhead in assertions.

[lhoestq]

lgtm :)