Support PyArrow arrays as tokenizer input

[mariosasko]

Most data processing libraries (Datasets, Polars, Pandas, DuckDB, etc.) are integrated with PyArrow, so native (zero-copy if possible) support for PyArrow arrays as input to avoid the unnecessary PyArrow to Python/NumPy conversion (pretty slow for string arrays) would be nice.

PS: PyArrow has recently added support for the PyCapsule interface, which should help with the implementation.

[ArthurZucker]

Would you like to open a PR for this? 🤗

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[mariosasko]

Marking this issue as a good first issue. If it doesn't get addressed after a while, I'll take a stab at it.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[WenheLI]

Interested in this. Can someone get it assigned to me?

[mariosasko]

@WenheLI Assigned :)

[WenheLI]

@mariosasko - Sorry just saw this! Can you guide me how to get started as I am still new to this project! Thanks a lot for your help!

[mariosasko]

Sure! The idea is to use the arrow crate (e.g., with ArrayData.from_pyarrow) to decode PyArrow StringArray/LargeStringArrays (when they are given as input to the Tokenizer). You can find the relevant code here (maybe this PR can also help, which has done the same thing for NumPy arrays).

To build the project, check this workflow file, in particular the part that installs the dependencies.

[shreya-51]

hello! @WenheLI are you still working on this?

[WenheLI]

@shreya-51 - Hi! Sorry for the late reply. And yes, I am still working on that

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.