Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add recipe for fine-tuning Zipformer with adapter #1512

Merged
merged 8 commits into from
Feb 29, 2024

Conversation

marcoyang1998
Copy link
Collaborator

@marcoyang1998 marcoyang1998 commented Feb 20, 2024

This PR supports fine-tuning a Zipformer transducer model with adapters (https://arxiv.org/pdf/1902.00751.pdf). The original model parameters are untouched and we only update the parameters in the adapters on the new domain.

The following table shows the WERs of different models on GigaSpeech test sets in a domain adaptation setting.

Model config trainable params dev test
pretrained (without fine-tuning) - 20.06 19.27
fine-tune whole model 66M 13.56 13.64
fine-tune with adapters 1.5M 15.05 15.18

By only introducing around 2% trainable parameters, the model with adapters achieves much lower WERs than the baseline model without adaptation.

@marcoyang1998
Copy link
Collaborator Author

After training, the model can be tested on both the original and the new domain.

To deactivate the adapters after training, set --use-adapters False. The performance on the original domain should be untouched.

@marcoyang1998 marcoyang1998 merged commit 7e2b561 into k2-fsa:master Feb 29, 2024
108 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants