Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use pytorch-labs/tokenizers and remove tokenizer/ #1401

Merged
merged 2 commits into from
Dec 7, 2024
Merged

Use pytorch-labs/tokenizers and remove tokenizer/ #1401

merged 2 commits into from
Dec 7, 2024

Conversation

larryliu0820
Copy link
Contributor

Summary: Use our shiny new repo
https://github.com/pytorch-labs/tokenizers.

Test Plan: Rely on CI jobs

Reviewers:

Subscribers:

Tasks:

Tags:

@larryliu0820
Use pytorch-labs/tokenizers and remove tokenizer/
449f0c8 
Summary: Use our shiny new repo
https://github.com/pytorch-labs/tokenizers.

Test Plan: Rely on CI jobs

Reviewers:

Subscribers:

Tasks:

Tags:
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Dec 6, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1401

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 45eb6ed with merge base 29428ef (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 6, 2024
@larryliu0820
Fix pull.yml
45eb6ed 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
Jack-Khuu
Jack-Khuu approved these changes Dec 7, 2024
View reviewed changes
Copy link
Contributor

@Jack-Khuu Jack-Khuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Outside of the linter, looks like the main change is UNWRAP to deref the tokenizer results. Does that sound right?

@larryliu0820
Copy link
Contributor Author

Outside of the linter, looks like the main change is UNWRAP to deref the tokenizer results. Does that sound right?

Yes exactly

@larryliu0820
Copy link
Contributor Author

All green merging

@larryliu0820 larryliu0820 merged commit fff956c into main Dec 7, 2024
54 checks passed
Jack-Khuu added a commit that referenced this pull request Dec 10, 2024
@Jack-Khuu
Revert "Use pytorch-labs/tokenizers and remove tokenizer/ (#1401)"
ba71ccd 
This reverts commit fff956c.
Jack-Khuu added a commit that referenced this pull request Dec 10, 2024
@Jack-Khuu
Revert "Use pytorch-labs/tokenizers and remove tokenizer/ (#1401)" (#…
3f9e66d 
…1414)

This reverts commit fff956c.
@Jack-Khuu Jack-Khuu mentioned this pull request Dec 23, 2024
Open
larryliu0820 added a commit that referenced this pull request Dec 26, 2024
@larryliu0820
[retry] Use pytorch-labs/tokenizers and remove tokenizer/ (#1401)
859a68d 
Summary: Retry of #1401

Test Plan: Re-run the repro command in #1413:

```
python3 torchchat.py generate llama3.2-1b-base --prompt "write me a story about a boy and his bear"

```

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820 larryliu0820 mentioned this pull request Dec 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jack-Khuu Jack-Khuu Jack-Khuu approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@larryliu0820 @Jack-Khuu @facebook-github-bot