Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DeepPavlov / transformer architecture token amount can't be predicted in advance #57

Closed
1 task done
nicolay-r opened this issue Apr 25, 2023 · 1 comment
Closed
1 task done

DeepPavlov / transformer architecture token amount can't be predicted in advance #57

nicolay-r opened this issue Apr 25, 2023 · 1 comment
Assignees
@nicolay-r
Labels
bug Something isn't working enhancement New feature or request wontfix This will not be worked on

Comments

@nicolay-r
Copy link
Owner

nicolay-r commented Apr 25, 2023 •
edited
Loading

Problem: exceeding limit of 512 tokens

  • add text chunking support for DeepPavlov NER pipeline item
@nicolay-r nicolay-r added bug Something isn't working wontfix This will not be worked on labels Apr 25, 2023
@nicolay-r nicolay-r self-assigned this Apr 25, 2023
nicolay-r added a commit that referenced this issue Apr 25, 2023
@nicolay-r
#57 enhanced
47200cb
@nicolay-r nicolay-r added the enhancement New feature or request label Apr 25, 2023
@nicolay-r
Copy link
Owner Author

Problem still exist, because we may only assume the potential size of input

nicolay-r added a commit that referenced this issue Apr 25, 2023
@nicolay-r
#57 upd, allow to modify chunk size
736f7ac
@nicolay-r nicolay-r changed the title DeepPavlov / transformer architecture token limitation can't be investigated in advance DeepPavlov / transformer architecture token limitation can't be predicted in advance Apr 25, 2023
@nicolay-r nicolay-r changed the title DeepPavlov / transformer architecture token limitation can't be predicted in advance DeepPavlov / transformer architecture token amount can't be predicted in advance Apr 25, 2023
nicolay-r added a commit that referenced this issue Apr 27, 2023
@nicolay-r
#57 catching the related exception
c670516
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nicolay-r nicolay-r

Labels
bug Something isn't working enhancement New feature or request wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nicolay-r