Abstract: The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation. To overcome these challenges, we explore strategies like fine-tuning and Chain-of-Thought prompting, demonstrating notable improvements. Our code and dataset are available at this link.

• [07/10/24] Our paper is accepted by COLM 2024!
• [02/29/24] Our dataset is available on 🤗huggingface!
• [02/02/24] Our paper is available on arxiv!

• Step 1: Set Up Environment

• Step 2: Download Dataset

• Step 3: Select MLLMs

  • Supported Models
  • Feature Your Own Model

• Step 4: Benchmark MLLMs

  • [Optional] Fine-Tuning Stage
  • Inference Stage
  • Evaluation Stage

• Step 5: Cite Us

To set up the environment for benchmarking MLLMs, please follow the following steps. This works for linux.

1. Clone this repository and rename it as cobsat

   git clone --recurse-submodules https://github.com/UW-Madison-Lee-Lab/CoBSAT
   mv CoBSAT cobsat
   cd cobsat

2. Install Packages

   Linux

   # create the environment that works for most of the cases
   conda create -n cobsat python=3.8.18
   conda activate cobsat
   pip install torch==2.1.2 torchvision==0.16.2 
   pip install -r conda_env/default_requirements.txt
   
   # create the environment for llava to work 
   conda create -n llava python=3.10.13
   conda activate llava
   pip install --upgrade pip  # enable PEP 660 support
   pip install git+https://github.com/yzeng58/LLaVA/@a61aae093656922fe16ec2152b031dd1de72fe92
   pip install -r conda_env/llava_requirements.txt
   
   # create the environment for gemini to work 
   conda env create -f conda_env/gemini.yml
   
   # create the environment for llava16 (LLaVA-NeXT) to work 
   conda env create -f conda_env/llava16.yml

   Mac

   TBA
   Windows

   TBA

3. Create environment.py in the cobsat directory. Note that many variables need you to config except root_dir on your own

   # Configure the environment variables for the project
   
   import os
   root_dir = os.path.dirname(os.path.abspath(__file__))
   
   SEED_PROJECT_ROOT = f'{root_dir}/models/SEED'
   
   ###############
   # NEED UPDATE #
   ###############
   TRANSFORMER_CACHE = '/data/yzeng58/.cache/huggingface/hub' 
   
   #########################
   # NEED UPDATE IF NEEDED #
   #########################
   # GPT-4V
   OPENAI_API_KEY = { 
     'key1': f'{your_openai_key_1}',
     'key2': f'{your_openai_key_2}',
   }
   # Gemini
   GEMINI_API_KEY = {
     'key1': f'{your_gemini_key_1}',
     'key2': f'{your_gemini_key_2}',
   }
   # Claude
   CLAUDE_API_KEY = {
     'key1': f'{your_claude_key_1}',
     'key2': f'{your_claude_key_2}',
   }
   # Emu for Image Generation
   EMU_IMAGE_PATH = '/data/yzeng58/cobsat/models/Emu/Emu1/model_weights/Emu/pretrain' 
   # Emu-Instruct
   EMU_INSTRUCT_PATH = '/data/yzeng58/cobsat/models/Emu/Emu1/model_weights/Emu/Emu-instruct.pt' 
   # Emu-Generation
   EMU_TEXT_PATH = '/data/yzeng58/cobsat/models/Emu/Emu1/model_weights/Emu/Emu-pretrain.pt'
   # WANDB Logging https://wandb.ai/site
   WANDB_ENTITY = 'lee-lab-uw-madison'
   WANDB_PROJECT = 'cobsat'

To use our dataset, please follow the following steps.

1. Download the images and their corresponding descriptions of our dataset.

   wget "https://huggingface.co/datasets/yzeng58/CoBSAT/resolve/main/datasets.zip"

2. Uncompress the datasets.zip file via unzip datasets.zip and move the datasets folder into your cobsat folder.

Up to now, the structure of your cobsat folder should look like this.

.
├── ...          
├── datasets                # download the dataset in this step
├── load_models
│   ├── call_emu.py
│   ├── call_emu2.py
│   ├── call_gill.py
│   ├── call_gpt.py
│   ├── call_llava.py       # LLaVA-1.5
│   ├── call_llava16.py     # LLaVA-NeXT 
│   ├── call_qwen.py
│   ├── call_seed.py
│   ├── call_gemini.py
│   ├── call_claude.py
│   ├── call_your_model.py  # [optional] create python file to load the model you want to evaluate
│   └── ... 
├── models                  
│   ├── SEED                
│   ├── gill                
│   ├── Emu                 
│   │   └── Emu1 
│   ├── LLaVA               
│   ├── Qwen-VL    
│   ├── Gemini
│   ├── Claude   
│   ├── OwnModel            # [optional] input your own model folder
│   └── ...
├── ...
├── environment.py          # follow the instruction above to create this file
├── load_model.py           # [optional] add your own model                
└── ...

We have implemented several state-of-the-art models for your convenience. Additionally, we offer guidelines for integrating your own MLLMs.

• SEED-LLaMA
  • Image Generation
  • Text Generation
  • Fine-Tuning
• GILL
  • Image Generation
  • Text Generation
• Emu
  • Image Generation
  • Text Generation
• Emu2
  • Image Generation
  • Text Generation

• GPT-4V
  • Text Generation
• LLaVA-1.5
  • Text Generation
• LLaVA-1.6/LLaVA-NeXT
  • Text Generation
• Qwen-VL
  • Text Generation
  • Fine-Tuning
• Gemini
  • Text Generation
• Claude
  • Text Generation

Throughout this section, the placeholder "OwnModel" can be substituted with the name of your specific model, such as "gpt4v".

1. Create your own model folder OwnModel/ in models/ if needed. Check this for examples.

2. Create python file call_OwnModel.py in load_models/ to load your own model.

   call_OwnModel.py template

   Your `call_OwnModel.py` script should include at least the following essential functions:

   • load_OwnModel: Utilized for loading the model to avoid repeated loading during inference or fine-tuning. In certain cases, this function may not be necessary. For example, OpenAI provides API access for GPT-4V, enabling inference without the need to explicitly load the model.
   • call_OwnModel: Employs the model to perform inference tasks.

   # Template
   def load_OwnModel(
       device = 'cuda',
       seed = 123,
   ):
   		...
       return model, others

   You have the flexibility to define the input parameters and the format of the return values according to your needs.

   # Template
   def call_OwnModel(
       text_inputs = ["Angry", "Cry", "Fly"],
       image_inputs = [
           "datasets/action_dog/angry_dog.jpg",
           "datasets/action_dog/cry_dog.jpg",
       ],
       seed = 123,
       gen_mode = 'text',
     	instruction = [
           "I will provide you with a few examples with text and images. Complete the example with the description of the next image. The description should be clear with main object, and include details such as color, texture, background, style, and action, if applicable. Tell me only the text prompt and I'll use your entire answer as a direct input to A Dalle-3. Never say other explanations. ",
           '',
       ],
     	call_mode = 'micl',
     	history = None,
     	save_history = False,
     	...
   ):
   		pass
       return output_dict

   • Necessary parameters for function call_OwnModel

     Below are the essential parameters required for configuring the model's operation.

     • call_mode: Specifies the operation mode with two options:
       • micl: Conducts text-to-image in-context learning (Multimodal In-Context Learning), combining text_inputs with image_inputs for model invocation.
       • text: Focuses solely on text_inputs, concatenating them for text-based processing.
     • text_inputs: A list of textual inputs used for demonstration and queries. Under micl mode, all but the last text_inputs are paired with image_inputs for in-context demonstrations. The final item is used as the textual query for model predictions.
     • image_inputs: A list of image inputs corresponding to text_inputs for multimodal demonstrations.
     • seed: An integer that sets the random seed, ensuring reproducibility of results.
     • gen_mode: Determines the type of output generated by the model:
       • text: Generates textual output.
       • image: Generate image output.
     • instruction: A list containing two strings designed to frame the prompt, with the first string placed at the beginning and the second at the end.
     • history: Tracks the history of previous prompts and model responses, aiding in multi-step reasoning. This parameter is optional (None by default). If utilized, the history should be integrated into new prompts.
     • save_history: Indicates whether to save the interaction history. This feature supports analyzing the model's learning and response patterns over time.

     Additional parameters may be used for customized usage.

   • Return values for function call_OwnModel

     The function returns a dictionary comprising the following keys.

     • description: Contains the generated textual output as a string.
     • image: Holds the generated image output. This key is populated only when gen_mode is set to image.
     • history: If the history argument of the function is set to True, this key stores the interaction history, including both prompts and responses.

     Additional content may be stored in custom keys to suit specialized requirements.

   Check call_seed.py or other files in load_models/ for concrete examples.

3. Add your own model in load_model.py.

   load_model.py template

       elif model == 'OwnModel':
           from load_models.call_OwnModel import load_OwnModel, call_OwnModel
   				... 
           return lambda configs: call_OwnModel(
               model, 
               others, 
               gen_mode = gen_mode, 
               **configs
           )

   You can check our implementation for other models for example. For your own model, typically you need to load the model first, and then use call_OwnModel to make the infernece. You may want to fix some parameters which are not related to the prompts such as model or tokenizer in this example. Here is one example usage:

   >>> from load_model import load_model
   >>> call_model = load_model('qwen')
   
   The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to "AutoModelForCausalLM.from_pretrained".
   Loading checkpoint shards: 100%|██████████| 10/10 [00:12<00:00,  1.21s/it]
   
   >>> call_model({
       'text_inputs': ["Red", "Green", 'Yellow'],
       'image_inputs': [
           "/data/yzeng58/micl/datasets/color_bag/red_bag.jpg",
           "/data/yzeng58/micl/datasets/color_bag/green_bag.jpg",
       ],
       'seed': 123,
   })
      
   {'description': 'The next image should be a yellow school bus on a white background, with the words "The Ultimate School Bus" in black font and "For all your school needs" in yellow font on the front. The bus should be a 3D model and the background should be a white field.',
    'time': 2.4913034439086914}

4. Add your model to supported_models in configs.py.

   supported_models = [
       'qwen', 
       'llava', 
       'llava16',
       'gpt4v', 
       'emu2', 
       'emu', 
       'seed',
       'gill',
       'gemini',
       'claude',
       'OwnModel', # your own model
   ]

5. Config the default instruction of your model by updating instruction_dict in configs.py.

   Especially, you need to edit the instruction_dict['default'].

   If your model support image generation, then you need to edit instruction_dict['default']['image'] .

   'image': {
       'gill': (
           'You are a professional assistant can generate a new image based on the seqeunce. ',
           '',
       ),
       ...
       # NEED UPDATE
       'OwnModel': (
           'Based on the sequence, generate the next image.',
           'Make the prediction now.'
       )
   }

   If your model support text generation, then you need to edit instruction_dict['default']['text'] .

   'text': {
       'seed': (
           "I will provide you a few examples with text and image. Complete the example with the description of next image. Tell me only the text prompt and I'll use your entire answer as a direct input to A Dalle-3. Never say other explanations. ",
           '',
       ),
       ...
       # NEED UPDATE
       'OwnModel': (
           'Based on the sequence, describe the next image clearly, including details such as the main object, color, texture, background, action, style, if applicable. ',
           'Make the prediction now.'
       )
   }

6. [Optional: If you want to finetune your model on our dataset] Update finetune_icl.py

   Please refer to the current example of fine-tuning Qwen-VL in here.

# Example
# activate the default environment
conda activate micl

python finetune_icl.py \
--model qwen \
--shot 2 \ 							
--prompt_type default \
--gen_mode text \
--ft_mode leave_one_out \
--eval_task_theme texture

The fine-tuned models will be stored in ft_models/.

# Example

# activate the default environment
# if the model is LLaVA, replace `micl` with `llava`
conda activate micl

python inference_icl.py \
--model seed \
--prompt_type default \
--gen_mode image \
--shot 2 4 \
--seed 123 \
--device cuda \
--task_id 1 2 3 \
--overwrite 0 \
--finetuned_model 0 \
--data_mode default \
--eval_mllm llava \
--ft_mode all \
--eval_task_theme ''

The generated outputs will be stored in results/exps/ by default or results/ft if finetuned_model is set to True.

# Example
# activate the environment for using LLaVA (since LLaVA is our evaluation model)
conda activate llava

python evaluation_icl.py \
--model seed \
--prompt_type default \
--eval_mode image \
--task_id 1 2 3 \
--shot 2 4 \
--device cuda \
--seed 123 \
--wandb 1 \
--overwrite 0 \
--finetuned_model 0 \
--data_mode default

The evaluation results will be stored in results/evals/ by default or results/ft if finetuned_model is set to True. If wandb is True, you can also view the evaluation results in your wandb board.

@article{zeng2024can,
  title={Can MLLMs Perform Text-to-Image In-Context Learning?},
  author={Zeng, Yuchen and Kang, Wonjun and Chen, Yicong and Koo, Hyung Il and Lee, Kangwook},
  journal={arXiv preprint arXiv:2402.01293},
  year={2024}
}