-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
481 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,45 @@ | ||
# LipGER | ||
Code for InterSpeech 2024 Paper: LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition | ||
|
||
|
||
## Training the LipGER model | ||
|
||
### 1. Prepare the checkpoint | ||
|
||
First prepare the checkpoints using: | ||
|
||
``` | ||
pip install huggingface_hub | ||
python scripts/download.py --repo_id meta-llama/Llama-2-7b-chat-hf --token your_hf_token | ||
python scripts/convert_hf_checkpoint.py --checkpoint_dir checkpoints/meta-llama/Llama-2-7b-chat-hf | ||
``` | ||
|
||
To see all available checkpoints, run: | ||
|
||
``` | ||
python scripts/download.py | grep Llama-2 | ||
``` | ||
|
||
For more details, you can also refer to [this link](https://github.com/YUCHEN005/RobustGER/tree/master/tutorials) | ||
|
||
### 2. Prepare the lip movements | ||
|
||
### 3. Fine-tune LipGER | ||
|
||
Then just run: | ||
|
||
``` | ||
sh finetune.sh | ||
``` | ||
|
||
### LipGER inference | ||
|
||
For inference just run: | ||
|
||
``` | ||
infer.sh | ||
``` | ||
|
||
|
Oops, something went wrong.