Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases

Install

cd Agri-LLaVA
conda create -n agrillava python=3.10 -y
conda activate agrillava
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Data

All of our data can be found in Huggingface

- Training data

Our Agri-400K consists of:

• Agricultural pests and diseases feature alignment data
• Agricultural pests and diseases instruction-tuning data

- Evaluation data

• Agri-LLaVA-Chatbot-Bench
• Agri-LLaVA-VQA-Bench

Train

Agri-LLaVA is trained on 8 A800 GPUs, with the entire training process taking 11 hours and 20 minutes.

- Stage 1: Agricultural Pre-training for Feature Alignment.

Hyperparameters

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
Agri-LLaVA	256	1e-3	1	2048	0

Pretrain

Please download the agricultural pests and diseases feature alignment data for pre-training from here

You can use the script to pre-train: pretrain.sh.

- Stage 2: Agricultural End-to-End Instruction-tuning.

Hyperparameters

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
Agri-LLaVA	128	2e-5	3	2048	0

Visual Instruction Tuning

Please download the agricultural pests and diseases instruction-tuning data for fine-tuning from here

You can use the following scripts to fine-tune:

• Full fine-tune: finetune.sh.
• Lora fine-tune: finetune_lora.sh.

Evaluation

Agricultural Multimodal Chatbot

- Prepare Data

Please download the Agri-LLaVA-Chatbot-Bench data for evaluating from here

- GPT-based Evaluation

1. Generate Agri-LLaVA responses on Agri-LLaVA-Chatbot-Bench.

python agri_llava/eval/model_vqa.py \
    --model-path /path/to/model_checkpoint \
    --question-file \
    /path/to/agri_llava_chatbot_question.jsonl \
    --image-folder \
    /path/to/image_folder \
    --answers-file \
    /path/to/agri_llava_chatbot_answer_our.jsonl

2. Evaluate the generated responses. In the command, agri_llava_chatbot_answer.jsonl refers to the reference answer generated by language-only GPT-4 based on knowledge.

python agri_llava/eval/eval_gpt_review_visual.py \
    --question /path/to/agri_llava_chatbot_question.jsonl \
    --answer-list \
    /path/to/agri_llava_chatbot_answer.jsonl \
    /path/to/agri_llava_chatbot_answer_our.jsonl \
    --rule agri_llava/eval/eval_metrics/rule.json \
    --output /path/to/review.jsonl

3. Summarize the evaluation results

python agri_llava/eval/summarize_gpt_review.py

Agricultural VQA

- Prepare Data

Please download the Agri-LLaVA-VQA-Bench data for evaluating from here.

- Fine-tuning on the VQA training data

We use the same scripts to adapt Agri-LLaVA to downstream tasks:

• Full fine-tune: finetune.sh.
• Lora fine-tune: finetune_lora.sh.

- Evaluating on the VQA testing data

1. Generate Agri-LLaVA responses on Agri-LLaVA-VQA-Bench.

python agri_llava/eval/model_vqa.py \
    --model-path /path/to/model_checkpoint \
    --question-file \
    /path/to/agri_llava_vqa_test_question.jsonl \
    --image-folder \
    /path/to/image_folder \
    --answers-file \
    /path/to/agri_llava_vqa_answer_our.jsonl

2. Evaluate the generated responses.

python agri_llava/eval/run_eval.py \
    --gt /path/to/agri_llava_vqa_test_answer.json \
    --pred /path/to/agri_llava_vqa_answer_our.jsonl

Acknowledgement

We sincerely appreciate LLaVA for making their models and code available as open-source contributions.