Checklist

What you can find in this repo:

• Selective EOS Supervision
  • training code
  • trained model checkpoints
• Scoring EOS Supervision
  • data filtering code
  • filtered data
• CHAIR evaluation
  • evaluation scripts
  • our test set data
• others
  • generated captions in Table 1

Selective EOS Supervision

Training

Following the instruction of LLaVA to prepare the environment, data (LLaVA-Instruction-150K) and pretraining models (e.g., LLaVA-1.5-7b).

Train the model with Selective EOS Supervision. The default configuration is set to train the llava-1.5-7b model with Detail23k for one epoch.

cd LLaVA
bash scripts/v1_5/selective_eos_finetune.sh

The main modifications to the original LLaVA code for Selective EOS Supervision are detailed in ./assets/selective-eos-supervision.md.

Checkpoint

Our models (LoRA weights) finetuned with Selective EOS Supervision:

Basic Model	Finetuning Data	Checkpoint
llava-1.5-7b	Detail23k	llava-v1.5-7b-selective-23k-lora
llava-1.5-7b	LLaVA-Instruction-150K	llava-v1.5-7b-selective-150k-lora

Scoring EOS Supervision

Data Scoring

For the LLaVA codebase, due to some constraints related to deepspeed, currently I have no idea about how to efficiently score a dataset with a standalone script. Our scoring relies on the training process, i.e., for each training step:

• Score the data in the minibatch and save the scores;
• Cancel loss backward (can be achieved by modifying the trainer code).

The core code for data scoring are provided in ./LLaVA/llava/model/language_model/llava_llama_filter.py.

Filtered Data

Our data filtered with Scoring EOS Supervision:

Basic Data	Filtered Data
LLaVA-Instruction-150K	LLaVA-Instruction-150K-filtered [OneDrive]

Training

Instruction tune the LLaVA-7b model on our filtered data with:

cd LLaVA
bash scripts/finetune_qlora_filtered.sh

CHAIR Evaluation

Data

The test set used in our paper for CHAIR evaluation is provided in ./CHAIR-eval/data/chair-500.jsonl. The data is randomly sampled from the MSCOCO validation set with a random seed of 0.

CHAIR Images

We provide two ways to collect test set images:

• a python script to collect images from the original MSCOCO images with softlinks. Please specify the path of your own MSCOCO image path. The script will create a folder ./CHAIR-eval/data/chair-500 for the CHAIR images.

  python ./CHAIR-eval/prepare_data.py

• a OneDrive link to download the 500 images. Unzip the images to ./CHAIR-eval/data/chair-500.

MSCOCO Annotation

Use the following command to download the annotation files of MSCOCO detection, which will be used for CHAIR evaluation:

cd ./CHAIR-eval/data
wget http://images.cocodataset.org/annotations/annotations_trainval2014.zip
mkdir MSCOCO
unzip -d MSCOCO/annotation annotations_trainval2014.zip

Evaluation

We provide a script for CHAIR inference and evaluation.
Set your model in the following script and then run it:

bash ./CHAIR-eval/eval.sh

MODEL_NAME: lora weights, e.g., yuezih/llava-v1.5-7b-selective-23k-lora
MODEL_BASE: base model checkpoint, e.g., liuhaotian/llava-v1.5-7b

The first-time evaluation can be slow because of the ground-truth object set construction. Subsequent evaluations will be faster with the cache.

Citation

If you find this repo helpful, please consider citing our paper:

@misc{yue2024less,
      title={Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective}, 
      author={Zihao Yue and Liang Zhang and Qin Jin},
      year={2024},
      eprint={2402.14545},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Acknowledgement

This repo is built on LLaVA (models) and OPERA (CHAIR evaluation). Many thanks for their efforts. The use of our code should also follow the original licenses.