Skip to content

Commit

Permalink
added some other details
Browse files Browse the repository at this point in the history
  • Loading branch information
Sreyan88 committed Jun 16, 2024
1 parent 4c77979 commit da6c371
Show file tree
Hide file tree
Showing 3 changed files with 481 additions and 0 deletions.
43 changes: 43 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,45 @@
# LipGER
Code for InterSpeech 2024 Paper: LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition


## Training the LipGER model

### 1. Prepare the checkpoint

First prepare the checkpoints using:

```
pip install huggingface_hub
python scripts/download.py --repo_id meta-llama/Llama-2-7b-chat-hf --token your_hf_token
python scripts/convert_hf_checkpoint.py --checkpoint_dir checkpoints/meta-llama/Llama-2-7b-chat-hf
```

To see all available checkpoints, run:

```
python scripts/download.py | grep Llama-2
```

For more details, you can also refer to [this link](https://github.com/YUCHEN005/RobustGER/tree/master/tutorials)

### 2. Prepare the lip movements

### 3. Fine-tune LipGER

Then just run:

```
sh finetune.sh
```

### LipGER inference

For inference just run:

```
infer.sh
```


Loading

0 comments on commit da6c371

Please sign in to comment.