Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tokenizer] set truncatation to default #3068

Merged
merged 1 commit into from
Apr 7, 2024

Conversation

frankfliu
Copy link
Contributor

Avoid crash when token exceed model max length.

Description

Brief description of what this PR is about

  • If this change is a backward incompatible change, why must this change be made?
  • Interesting edge cases to note here

@frankfliu frankfliu requested review from zachgk and a team as code owners April 7, 2024 01:34
Avoid crash when token exceed model max length.
@codecov-commenter
Copy link

codecov-commenter commented Apr 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.29%. Comparing base (ae00167) to head (ec0959d).
Report is 1 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff            @@
##             master    #3068   +/-   ##
=========================================
  Coverage     72.28%   72.29%           
- Complexity     7302     7303    +1     
=========================================
  Files           722      722           
  Lines         32593    32592    -1     
  Branches       3410     3410           
=========================================
+ Hits          23561    23563    +2     
  Misses         7407     7407           
+ Partials       1625     1622    -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@frankfliu frankfliu merged commit 8bd3bf6 into deepjavalibrary:master Apr 7, 2024
5 checks passed
@frankfliu frankfliu deleted the truncate branch April 7, 2024 19:07
frankfliu added a commit that referenced this pull request Apr 26, 2024
Avoid crash when token exceed model max length.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants