Skip to content

【AAAI2025】MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt

License

Notifications You must be signed in to change notification settings

924973292/MambaPro

Repository files navigation

MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt

Description of the image

Yuhao Wang · Xuehu Liu · Tianyu Yan · Yang Liu · Aihua Zheng · Pingping Zhang* · Huchuan Lu

AAAI 2025 Paper

Overall Framework

MambaPro is a novel multi-modal object ReID framework that integrates CLIP's pre-trained capabilities with state-of-the-art multi-modal aggregation techniques. Using Parallel Feed-Forward Adapters (PFA), Synergistic Residual Prompts (SRP), and the innovative Mamba Aggregation (MA) mechanism, it achieves robust performance with reduced computational complexity. MambaPro sets new standards in handling long sequences and missing modalities.


News

  • We released the MambaPro codebase and paper! 🚀 Paper
  • Great news! Our paper has been accepted to AAAI 2025! 🎉

Table of Contents


Introduction

Multi-modal object ReID leverages complementary data from diverse modalities (e.g., RGB, NIR, TIR) to overcome challenges like poor lighting and occlusion. MambaPro advances this field by:

  1. PFA: Transferring CLIP's pre-trained knowledge to ReID tasks via parallel adapters.
  2. SRP: Integrating modality-specific prompts with synergistic transformations.
  3. MA: Efficiently modeling intra- and inter-modality interactions with linear complexity.

Contributions

  • Introduced MambaPro, the first CLIP-based framework for multi-modal object ReID.
  • Developed SRP for synergistic learning across modalities with residual refinements.
  • Proposed MA, achieving linear complexity for long-sequence multi-modal interactions.
  • Validated effectiveness on RGBNT201, RGBNT100, and MSVR310 benchmarks.

Results

Multi-Modal Object ReID

Multi-Modal Person ReID [RGBNT201]

RGBNT201 Results

Multi-Modal Vehicle ReID [RGBNT100 & MSVR310]

Vehicle Results

Ablation Studies [RGBNT201]

Ablation Results

Hyperparameter Analysis [RGBNT201]

Hyperparameter Analysis


Visualizations

Feature Distribution (t-SNE)

t-SNE Distribution

GradCam Visualization

Prompt Comparisons

Prompt Comparisons


Reproduction

Datasets

Pretrained Models

Configuration

  • RGBNT201: configs/RGBNT201/MambaPro.yml
  • RGBNT100: configs/RGBNT100/MambaPro.yml
  • MSVR310: configs/MSVR310/MambaPro.yml

Training

#!/bin/bash
# python = 3.10.13
# cuda = 11.8
source activate (your_env)
cd (your_path)
pip install -r requirements.txt
cd selective_scan && pip install .
python train_net.py --config_file configs/RGBNT201/MambaPro.yml

Note

  • If you want to use a CLIP-based framework for multi-modal object ReID, DeMo is a better choice, the prompt/adapter tuning configuration in MambaPro is retained for users, besides, we provide detailed visualizations code in DeMo.
  • Thanks for your attention and support!

Star History

Star History Chart


Citation

If you find MambaPro helpful in your research, please consider citing:

@inproceedings{wang2025MambaPro,
  title={MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt},
  author={Wang, Yuhao and Liu, Xuehu and Yan, Tianyu and Liu, Yang and Zheng, Aihua and Zhang, Pingping and Lu, Huchuan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025}
}

About

【AAAI2025】MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published