IterableDataset formatting in numpy/torch/tf/jax

[lhoestq]

This code now returns a numpy array:

from datasets import load_dataset

ds = load_dataset("imagenet-1k", split="train", streaming=True).with_format("np")
print(next(iter(ds))["image"])

It also works with "arrow", "pandas", "torch", "tf" and "jax"

Implementation details:

I'm using the existing code to format an Arrow Table to the right output format for simplicity.
Therefore it's probbaly not the most optimized approach.

For example to output PyTorch tensors it does this for every example:

python data -> arrow table -> numpy extracted data -> pytorch formatted data

Releasing this feature

Even though I consider this as a bug/inconsistency, this change is a breaking change.
And I'm sure some users were relying on the torch iterable dataset to return PIL Image and used data collators to convert to pytorch.

So I guess this is datasets 3.0 ?

TODO

• merge Image & Audio formatting for numpy/torch/tf/jax #5072
• docs
• tests

Close #5083

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[lhoestq]

Actually I'm not happy with this implementation. It always require the iterable dataset to have definite features, which removes a lot of flexibility. So I think we need an actual formatting from python objects, not from arrow data.

[lhoestq]

Closing this one since it has too many conflicts and still require some work - it will be easier to open a new PR