SignGen: End-to-End Sign Language Video Generation with Latent Diffusion

Method

Experiment Results

RWTH-2014

RWTH-2014T

AUTSL

Running by Yourself

1. Installation

create a conda environment.

conda create -n  xxx  python==3.8.5 

Then you can create the same environment as ours with the following command:

 pip install -r requirements.txt # install all requirements 

2. Download model weights

For LPIPS

The code will do it for you!

Code will download Alex and move it into: models/weights/v0.1/alex.pth

For FVD

The code will do it for you!

Code will download i3D model pretrained on Kinetics-400 Use models/fvd/convert_tf_pretrained.py to make i3d_pretrained_400.pt

3. Datasets

You can download these datasets such as RWTH-2014,RWTH-2014T and AUTSL.

How the data was processed:

1. Download AUTSL dataset to /path/to/AUTSL:\
2. Convert 128x128 images to HDF5 format:
   `python datasets/sign_language_convert.py --sl_dir 'datasets/videos' --split 'train' --out_dir 'datasets/signLanguages/train' --image_size 128 --force_h5 False

Training -The code is coming soon.