Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Llava chat template examples #30130

Merged
merged 1 commit into from
Apr 11, 2024
Merged

Fix Llava chat template examples #30130

merged 1 commit into from
Apr 11, 2024

Conversation

lewtun
Copy link
Member

@lewtun lewtun commented Apr 8, 2024 •
edited
Loading

What does this PR do?

This PR fixes a few minor issues in the chat templating examples for llava models. In particular, looking at the inference code from the original Llava repo, one sees that the correct template is:

USER: <image>\n<prompt_1> ASSISTANT:<response_1></s>USER: <prompt_2> ASSISTANT:<response_2> etc

while we had:

USER: <image>\n<prompt_1> \nASSISTANT:<response_1>\nUSER: <prompt_2>\nASSISTANT:<response_2> etc

I've only fixed the llava-v1.5 parts of the code, but happy to extend the PR to cover the other models if we agree with the change.

Code to reproduce the expected template from the Llava repo:

# pip install git+https://github.com/haotian-liu/LLaVA.git
>>> from llava.conversation import conv_templates
>>> conv = conv_templates["llava_v1"].copy()
>>> conv.append_message(conv.roles[0], "<image>\nHello")
>>> conv.append_message(conv.roles[1], "How are you?")
>>> conv.append_message(conv.roles[0], "Good")
>>> conv.append_message(conv.roles[1], "Yay")
>>> conv.get_prompt()
"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nHello ASSISTANT: How are you?</s>USER: Good ASSISTANT: Yay</s>"

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc @NielsRogge @amyeroberts

lewtun
lewtun commented Apr 8, 2024
View reviewed changes
src/transformers/models/llava/modeling_llava.py
@@ -388,16 +389,16 @@ def forward(
>>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf")
>>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")

>>> prompt = "<image>\nUSER: What's the content of the image?\nASSISTANT:"
>>> prompt = "USER: <image>\nWhat's the content of the image? ASSISTANT:"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT this prompt put the <image> token in the wrong place altogether.

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lewtun
lewtun commented Apr 8, 2024
View reviewed changes
docs/source/en/model_doc/llava.md
@@ -43,13 +43,13 @@ The original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/
- For better results, we recommend users to prompt the model with the correct prompt format:

```bash
"USER: <image>\n<prompt>ASSISTANT:"
"USER: <image>\n<prompt> ASSISTANT:"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically we should prepend the system prompt here, but I think it can be excluded for the purposes of keeping the demo simple.

NielsRogge
NielsRogge approved these changes Apr 10, 2024
View reviewed changes
Copy link
Contributor

@NielsRogge NielsRogge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing! FYI @ArthurZucker @younesbelkada

amyeroberts
amyeroberts approved these changes Apr 10, 2024
View reviewed changes
Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing these!

src/transformers/models/llava/modeling_llava.py
>>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
"\nUSER: What's the content of the image?\nASSISTANT: The image features a stop sign on a street corner"
"USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we expecting to have this double spacing here "USER: \nWhat's ? We're skipping <image> but from the prompt I would have expected: USER: \nWhat's

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I did find this a bit weird, but it seems to be the default behaviour of the decoding

@lewtun lewtun merged commit fbdb978 into main Apr 11, 2024
18 checks passed
@lewtun lewtun deleted the fix-llava-doc branch April 11, 2024 08:38
ArthurZucker pushed a commit that referenced this pull request Apr 22, 2024
@lewtun @ArthurZucker
Fix Llava chat template examples (#30130)
5940428
itazap pushed a commit that referenced this pull request May 14, 2024
@lewtun
Fix Llava chat template examples (#30130)
8c3019d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NielsRogge NielsRogge NielsRogge approved these changes

@amyeroberts amyeroberts amyeroberts approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lewtun @HuggingFaceDocBuilderDev @amyeroberts @NielsRogge