Replace GPT2TokenizerFast with tiktoken-based tokenizer (#44)#44

Closed

github-actions[bot] wants to merge 4 commits intomainfrom autopr/43

+40-13

Commits on Apr 11, 2023

Implement tiktoken-based tokenizer in autopr.utils.tokenizer
AutoPR
committed
Update tokenizer usage in autopr/utils/repo.py
AutoPR
committed
Update tokenizer usage in autopr/repos/completions_repo.py
AutoPR
committed
Add tests for the new tiktoken-based tokenizer
AutoPR
committed