-
[2024.12.15] 🔥🔥 Our DCE pipeline is launching soon—stay tuned !
-
[2024.12.15] 🔥🤗 Our DCE-1M datasets is released, please check out and download in EDC-1M !
DCE leverage visual specialists to replicate various human visual capabilities, and subsequently employ large language models (LLMs) to simulate the human cognitive process. This combined approach enables us to generate high-quality image captions by closely mimicking the way humans perceive and interpret visual information.
-
Open-Source Accessibility: The DCE pipeline is built entirely using open-source models, providing an accessible, cost-effective solution for generating high-quality image captions without reliance on proprietary technologies.
-
Customizable and Flexible Design: The pipeline supports a DIY approach, allowing users to integrate and combine different visual specialist models tailored to their specific needs. This flexibility empowers users to generate customized captions enriched with targeted visual information.
Training Large Multimodality Models (LMMs)s relies on descriptive image caption that connects image and language. Existing methods either distill the caption from the LMM models or construct the captions from the internet images or by human. We propose to leverage off-the-shelf visual specialists, which were trained from annotated images initially not for image captioning, for enhancing the image caption.
Our approach, named DCE, explores object low-level and fine-grained attributes (e.g., depth, emotion and fine-grained categories) and object relations (e.g., relative location and human-object-interaction (HOI)), and combine the attributes into the descriptive caption. Experiments demonstrate that such visual specialists are able to improve the performance for visual understanding tasks as well as reasoning that benefits from more accurate visual understanding. We will release the source code and the pipeline so that other visual specialists are easily combined into the pipeline.
