Some questions about finetuning and LISA++

[chris5zk]

Hello,
Your work is truly impressive! I’m particularly interested in the instance segmentation performance presented in the LISA++ paper.
I’m trying to reproduce the LISA++ results using the pretrained LISA model with my own data, but I’ve encountered the following issues:

1. Fine-tuning LISA with my own data:
   When using LISA as a pretrained model to fine-tune on my data, I ran into the error:
   AttributeError: 'LISAForCausalLM' object has no attribute 'ce_loss_weight'.
   Thanks to the solution provided in AttributeError: 'LISAForCausalLM' object has no attribute 'ce_loss_weight' #85 , modifying the LISA.py file by replacing:

   if not hasattr(config,  train_mask_decoder"):

   with

   if hasattr(config, "train_mask_decoder"):

   then the code now runs successfully. However, I’m unsure why this error occurs. Could you clarify if this adjustment might cause any issues, or if it’s simply a typo in the code?

2. Instruction Data formatting and tokenizer compatibility:
   Following the instructions in the LISA++ paper, I formatted the reasoning problem answers as class_name [ID] to serve as instruction data. However, this seems a bit unreasonable since the tokenizer only includes the [SEG] token. I noticed that in LISA.py, the [SEG] IDs are counted to decide whether to generate a prompt for the mask decoder. This causes a mismatch between gt_mask and pred_mask. Are there additional details I should be aware of to avoid such issues? or should I add an [SEG] behind the class ID?

Will you be providing more details or guidance on this? I would deeply appreciate any help or clarification on these matters.

Thank you so much!