Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Packing for pretokenised #468

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: Packing for pretokenised #468

wants to merge 1 commit into from

Conversation

dushyantbehl
Copy link
Collaborator

Description of the change

This PR adds support for packing for pretokenized datasets which was added in transformers>=4.46

This is based on the PR #448

Related issue number

How to verify the PR

Was the PR tested

  • I have added >=1 unit test(s) for every new method I have added.
  • I have ensured all unit tests pass

@github-actions GitHub Actions
Copy link

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

@github-actions github-actions bot added the feat label Feb 13, 2025
@dushyantbehl dushyantbehl force-pushed the packing-for-pretokenised branch from d6bcee1 to 4c3f1c3 Compare February 13, 2025 12:19
@dushyantbehl dushyantbehl mentioned this pull request Feb 13, 2025
Merged
2 tasks
@dushyantbehl dushyantbehl force-pushed the packing-for-pretokenised branch from 4c3f1c3 to d0e0e20 Compare February 13, 2025 12:45
@kmehant
Copy link
Collaborator

kmehant commented Feb 13, 2025 •
edited
Loading

@dushyantbehl

  1. max_seq_len is too long in your testcase that it was not able to generate not even 1 sample so it was failing
  2. you should pass seq2seq collator with padding=False when it is pretokenized but it seem to pick up completiononly collator.

@willmj willmj mentioned this pull request Feb 20, 2025
Merged
5 tasks
@dushyantbehl dushyantbehl force-pushed the packing-for-pretokenised branch 4 times, most recently from 0c589bd to 6d4e771 Compare March 3, 2025 11:26
@dushyantbehl
Add seq2seq collator for packing pretokenized data
73b81c1 
Signed-off-by: Dushyant Behl <dushyantbehl@in.ibm.com>
@dushyantbehl dushyantbehl force-pushed the packing-for-pretokenised branch from 6d4e771 to 73b81c1 Compare March 3, 2025 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anhuong anhuong Awaiting requested review from anhuong anhuong is a code owner

@Ssukriti Ssukriti Awaiting requested review from Ssukriti Ssukriti is a code owner

@aluu317 aluu317 Awaiting requested review from aluu317 aluu317 is a code owner

@fabianlim fabianlim Awaiting requested review from fabianlim fabianlim is a code owner

@kmehant kmehant Awaiting requested review from kmehant kmehant is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
feat
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dushyantbehl @kmehant