Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an AlbertMaskedLM task model #718

Closed
4 tasks
mattdangerw opened this issue Feb 3, 2023 · 1 comment · Fixed by #725
Closed
4 tasks

Add an AlbertMaskedLM task model #718

mattdangerw opened this issue Feb 3, 2023 · 1 comment · Fixed by #725
Assignees
@shivance
Labels
stat:contributions welcome Add this label to feature request issues so they are separated out from bug reporting issues type:feature New feature or request

Comments

@mattdangerw
Copy link
Member

mattdangerw commented Feb 3, 2023 •
edited
Loading

In #653 we added a masked language modeling task for RoBERTa. We can make a similar change for the Albert model.

  • Update AlbertTokenizer to expect a mask token.
  • Add a AlbertMaskedLMPreprocessor preprocessor layer and tests.
  • Add a AlbertMaskedLM task model and tests.
  • Update keras_nlp/models/__init__.py to export AlbertMaskedLM and AlbertMaskedLMPreprocessor.
@mattdangerw mattdangerw added type:feature New feature or request stat:contributions welcome Add this label to feature request issues so they are separated out from bug reporting issues labels Feb 3, 2023
@shivance
Copy link
Collaborator

shivance commented Feb 3, 2023

I would love to work on this issue!

@shivance shivance mentioned this issue Feb 3, 2023
Closed
4 tasks
@shivance shivance mentioned this issue Feb 5, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shivance shivance

Labels
stat:contributions welcome Add this label to feature request issues so they are separated out from bug reporting issues type:feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adding an AlbertMaskedLM task + Fix Projection layer dimension in MaskedLMHead shivance/keras-nlp
2 participants
@mattdangerw @shivance