Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for pyarrow arrays as input #1535

Closed
wants to merge 7 commits into from

Conversation

notjedi
Copy link

@notjedi notjedi commented May 18, 2024

closes #1415

@notjedi
Copy link
Author

notjedi commented May 18, 2024

yet to add support for pyarrow.LargeString

@notjedi
Copy link
Author

notjedi commented May 18, 2024

pulling arrow from git because the current stable version v51.0 links against v0.20 of pyo3 while the bindings link against v0.21. there is already a merged pull request(apache/arrow-rs#5566) which migrates from v0.20 to v0.21 of pyo3 which i think will be included in the next release of v52.0.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can leave the dep as optional wdyt?

bindings/python/Cargo.toml Outdated Show resolved Hide resolved
bindings/python/src/tokenizer.rs Show resolved Hide resolved
@notjedi
Copy link
Author

notjedi commented Jun 7, 2024

I think we can leave the dep as optional wdyt?

makes sense, will get it changed @ArthurZucker

@notjedi
Copy link
Author

notjedi commented Jun 7, 2024

i've added pyarrow as a feature in pyproject.toml. not sure if that is what we want here. let me know if i need to fix anything else. @ArthurZucker

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want to avoid having to use git for the repo 😢
ALso do you have any bench / examples of it being use and fast? 🚀

@notjedi
Copy link
Author

notjedi commented Jun 7, 2024

yeah will update that once there is a stable release of arrow (v52.0). not really, i don't have any benchmarks and the only example i have is in the tests. i don't need this personally, just wanted to close an open issue and contribute.

@strategy155
Copy link

Good afternoon everyone. How do you think the optimal benchmark should look like? Native pyarrow vs numpy conversion for string arrays?

@ArthurZucker
Copy link
Collaborator

something like that yeah, I have no idea in what context pyarrow is used as I have not used it, but in optimal context usage

@github-actions github-actions bot added the Stale label Jul 12, 2024
@github-actions github-actions bot closed this Jul 17, 2024
@strategy155
Copy link

I'm not sure it should be closed. The pr is looking fine.

@notjedi
Copy link
Author

notjedi commented Jul 17, 2024

yes, just update to arrow to the latest stable version v52.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support PyArrow arrays as tokenizer input
4 participants