Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Tokenizers] Port BERTTokenizers #6991

Open
ericstj opened this issue Feb 8, 2024 · 1 comment
Open

[Tokenizers] Port BERTTokenizers #6991

ericstj opened this issue Feb 8, 2024 · 1 comment
Labels
enhancement New feature or request P1 Priority of the issue for triage purpose: Needs to be fixed soon.
Milestone
ML.NET 4.0

Comments

@ericstj
Copy link
Member

ericstj commented Feb 8, 2024 •
edited
Loading

Porting BERTTokenizers enables several text embedding generation models. Requires #6988.

https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#text-embeddings.
https://github.com/huggingface/transformers/blob/v4.37.0/src/transformers/models/bert/tokenization_bert.py#L137

cc @luisquintanilla

We already have some BERT implementation which may be sufficient.
@ericstj ericstj added enhancement New feature or request P2 Priority of the issue for triage purpose: Needs to be fixed at some point. labels Feb 8, 2024
@ericstj ericstj added this to the ML.NET Future milestone Feb 8, 2024
@ericstj ericstj mentioned this issue Feb 8, 2024
Open
13 tasks
@luisquintanilla luisquintanilla mentioned this issue Mar 28, 2024
Merged
@ericstj ericstj modified the milestones: ML.NET Future, ML.NET 4.0 Aug 19, 2024
@ericstj ericstj added P1 Priority of the issue for triage purpose: Needs to be fixed soon. and removed P2 Priority of the issue for triage purpose: Needs to be fixed at some point. labels Aug 19, 2024
@ds5678
Copy link

ds5678 commented Nov 13, 2024

Has this been implemented? It seems very similar to #7281.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request P1 Priority of the issue for triage purpose: Needs to be fixed soon.
Projects
None yet
Milestone
ML.NET 4.0
Development

No branches or pull requests
2 participants
@ericstj @ds5678