Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow fewshots for multimodal tasks #2450

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

artemorloff
Copy link
Contributor

till now only zero-shot tasks may bne processed

if any of the few-shots include images, they are not passed into the model. This behavior causes errors

I suggest processing the images while constructing the task. Fewshot sampler generates labeled_examples and multimodal_args. For future multimodal_args may include any modality. Now only images

@artemorloff
Copy link
Contributor Author

@baberabb @haileyschoelkopf @lintangsutawika tests fail due to changes in fewshot_context function
it now returns not only ctx, but also multimodal_args - a dict with all files the come with these fewshots

I have checked the code on mmmu (qwen2-vl-2b) and mmlu (to ensure text-only tasks are not affected). Do not see any difference in the output

@artemorloff
Copy link
Contributor Author

@baberabb @haileyschoelkopf if you like the general idea, i could modify the tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant