VisionZip: Longer is Better but Not Necessary in Vision Language Models

TABLE OF CONTENTS

1. News
2. Highlights
3. Video
4. Demo
5. Installation
6. Quick Start
7. Evaluation
8. Examples
9. Citation
10. Acknowledgement
11. License

News

• [2024.12.16] Due to positive feedback on our demo, we have released the VisionZip Demo-Chat code in a new branch, 'demo-chat'.
• [2024.12.05] We add an Usage-Video, providing a step-by-step guide on how to use the demo.
• [2024.12.05] We add a new Demo-Chat, where users can manually select visual tokens to send to the LLM and observe how different visual tokens affect the final response. We believe this will further enhance the analysis of VLM interpretability.
• [2024.11.30] We release Paper and this GitHub repo, including code for LLaVA.

VisionZip: Longer is Better but Not Necessary in Vision Language Models [Paper]
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, Jiaya Jia

Highlights

1. Our VisionZip achieves state-of-the-art performance among efficient VLM methods. By retaining only 10% of visual tokens, it achieves nearly 95% of the performance in training-free mode.
2. VisionZip can be applied during the inference stage (without incurring any additional training cost), the efficient tuning stage (to achieve better results), and the training stage (almost no performance degradation，saving 2× memory and 2× training time).
3. VisionZip significantly reduces the prefilling time and the total inference time (with KV cache enabled).
4. Why does this simple, text-agnostic method outperform text-relevant methods? We conduct an in-depth analysis in the paper and provide a demo to visualize these findings.
5. Since VisionZip is a text-agnostic method that reduces visual tokens before input into the LLM, it can adapt to any existing LLM acceleration algorithms and is applicable to any task that a vanilla VLM can perform, such as multi-turn conversations.

Video

Demo

Speed Improvement

The input video is about the Titanic, and the question is, "What’s the video talking about?"

It is important to note that the left side shows the vanilla model, which encodes only 16 frames, while the right side shows our VisionZip, which, despite encoding 32 frames, is still twice as fast as the vanilla model.

Visualize Redundancy and Misalignment

Explore the visual redundancy and feature misalignment in the above Demo. To run it locally, use the following command:

python gradio_demo.py 

Observe How Different Visual Tokens Impact the Final Response

This Demo-Chat lets users to manually select which visual tokens to send to the LLM and observe how different visual tokens affect the final response.

Installation

Our code is easy to use.

1. Install the LLaVA environment.

2. For formal usage, you can install the package from PyPI by running the following command:

pip install visionzip

For development, you can install the package by cloning the repository and running the following command:

git clone https://github.com/dvlab-research/VisionZip
cd VisionZip
pip install -e .

Quick Start

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
from llava.eval.run_llava import eval_model
from visionzip import visionzip
model_path = "liuhaotian/llava-v1.5-7b"

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)
## VisoinZip retains 54 dominant tokens and 10 contextual tokens
model = visionzip(model, dominant=54, contextual=10)

Evaluation

The evaluation code follows the structure of LLaVA or Lmms-Eval. After loading the model, simply add two lines as shown below:

## Load LLaVA Model (code from llava.eval.model_vqa_loader)
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name)
## add VisionZip
from visionzip import visionzip
model = visionzip(model, dominant=54, contextual=10)

Examples

Multi-turn Conversations

VisionZip, which extracts text-agnostic tokens, is better suited for multi-turn dialogue.

Longer Videos with More Frames

VisionZip reduces the number of visual tokens per frame, allowing more frames to be processed. This improves the model's ability to understand longer videos.

Citation

If you find this project useful in your research, please consider citing:

@article{yang2024visionzip,
  title={VisionZip: Longer is Better but Not Necessary in Vision Language Models},
  author={Yang, Senqiao and Chen, Yukang and Tian, Zhuotao and Wang, Chengyao and Li, Jingyao and Yu, Bei and Jia, Jiaya},
  journal={arXiv preprint arXiv:2412.04467},
  year={2024}
}

Acknowledgement

• This work is built upon LLaVA, mini-Gemini, Lmms-Eval, and Video-LLaVA. We thank them for their excellent open-source contributions.

• We also thank StreamingLLM, FastV, SparseVLM, and others for their contributions, which have provided valuable insights.

License

• VisionZip is licensed under the Apache License 2.0.