New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ErnieGramTokenizer可以添加自定义词典吗？ #2022

Closed

wjddd opened this issue Apr 27, 2022 · 1 comment

Assignees

wjddd commented Apr 27, 2022

欢迎您反馈PaddleNLP使用问题，非常感谢您对PaddleNLP的贡献！
在留下您的问题时，辛苦您同步提供如下信息：

版本、环境信息
1）PaddleNLP和PaddlePaddle版本：请提供您的PaddleNLP和PaddlePaddle版本号，例如PaddleNLP 2.0.4，PaddlePaddle2.1.1
2）系统环境：请您描述系统类型，例如Linux/Windows/MacOS/，python版本
复现信息：如为报错，请给出复现环境、复现步骤

linjieccc assigned smallv0221

Collaborator

wawltor commented Apr 29, 2022

您好，目前ErnieGramTokenizer是不能自定义词典的，因为在预训练时词典已经固定的，同时在词典中没有留无用占位给新增的Token，随便改变词典会影响效果；想问一下，自定义词典目的是什么了？

wjddd closed this as completed

NatLee mentioned this issue

自定義vocab.txt #2649

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment