AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM

🛠️ Installation

Clone this repository:

git clone https://github.com/wangjiarui153/AIGV-Assessor.git

Create a conda virtual environment and activate it:

conda create -n aigvassessor python=3.9 -y
conda activate aigvassessor

Install dependencies using requirements.txt:

pip install -r requirements.txt

🌈 Training

for stage1 training (Spatiotemporal Projection Module)

sh shell/train/stage1_train.sh

for stage2 training (Fine-tuning the vision encoder and LLM with LoRA)

sh shell/train/stage2_train.sh

🌈 Evaluation

for stage1 evaluation (Text-based quality levels)

sh shell/eval/stage1_eval.sh

for stage2 evaluation (Scores from 4 perspectives)

sh shell/eval/stage2_eval.sh

📌 TODO

• ✅ Release the training code (stage1 and stage2)
• ✅ Release the evaluation code (stage1 and stage2)
• Release the AIGVQA-DB

📧 Contact

If you have any inquiries, please don't hesitate to reach out via email at wangjiarui@sjtu.edu.cn

🎓Citations

If you find AIGV-Assessor is helpful, please cite:

@misc{wang2024aigvassessorbenchmarkingevaluatingperceptual,
      title={AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM}, 
      author={Jiarui Wang and Huiyu Duan and Guangtao Zhai and Juntong Wang and Xiongkuo Min},
      year={2024},
      eprint={2411.17221},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.17221}, 
}