- compare-bpe-tiktoken.ipynb benchmarks various byte pair encoding implementations
- bpe_openai_gpt2.py is the original bytepair encoder code used by OpenAI
02_bonus_bytepair-encoder
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
parent directory.. | ||||
| Name | Name | Last commit date | ||
|---|---|---|---|---|
parent directory.. | ||||