The caption generated using “Start_caption_with” does not seem to be as expected

[YuanGYao]

Hi, I found that after using "start_caption_with", the caption output by the model became different.

After checking the code, I found that in the "get_model_inputs" method, you merged "start_caption_with" and "prompt" and passed them to the model together. Is this something you did intentionally?

What I expected for "start_caption_with" would be to do what you did in the "get_caption_from_generated_tokens" method and simply add the contents of "start_caption_with" to the beginning of the contents put back by the model.

[jhc13]

This is the intended behavior so that the captioning model can generate a natural continuation of the Start caption with text.

For example, if Start caption with is set to photo of a person wearing (with the intent of describing the subject's clothing), the model might generate a caption like photo of a person wearing a T-shirt and jeans ....

If this was instead done like the way you described, the model might initially generate a caption like This image shows a man ..., and when the Start caption with text is added, it would become photo of a person wearing This image shows a man ..., which is unnatural and ignores the intent of the Start caption with text.

[YuanGYao]

Haha, it seems that my previous understanding was wrong.

When I used A1111's wd14 tagger extension or other tagging plug-ins before, it also had a similar prefix text box, but it just directly placed the text in the text box at the front of the model output tag. So I expected the same behavior from TagGUI as well.

However, I think if you want the model to start with a certain paragraph of text when outputting text, it may be more reasonable to describe it directly in the prompt.

[jhc13]

When I used A1111's wd14 tagger extension or other tagging plug-ins before, it also had a similar prefix text box, but it just directly placed the text in the text box at the front of the model output tag. So I expected the same behavior from TagGUI as well.

I see. I haven't used those tools, so I'm not familiar with their behavior.

However, I think if you want the model to start with a certain paragraph of text when outputting text, it may be more reasonable to describe it directly in the prompt.

That can work sometimes. Start caption with can be helpful in cases where the prompt is not enough.

[YuanGYao]

I see. Thanks for your explanation.