Add LMMs-Lite #148
Merged
Add LMMs-Lite #148
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Luodian
added a commit
that referenced
this pull request
Sep 1, 2024
* feat: Add multi-choice parsing and processing functions This commit adds two new functions to the `egoschema/utils.py` file: `get_multi_choice_info` and `parse_multi_choice_response`. These functions are used to parse and process multi-choice responses in the Egoschema task. The `get_multi_choice_info` function extracts information about the available choices from the input document, while the `parse_multi_choice_response` function parses the generated response and returns the predicted index. These functions are essential for accurately processing multi-choice answers in the Egoschema task. * feat: Add regex-based parsing for multi-choice predictions This commit enhances the `perceptiontest_val_process_results_mc` function in the `utils.py` file. It introduces regex-based parsing to extract the predicted choice from the raw text prediction. If a match is found for A, B, C, or D, the matched letter is used as the prediction. Otherwise, an empty string is set as the prediction. This improvement ensures accurate processing of multi-choice predictions in the perception test validation. Co-authored-by: [Co-author Name] <[coauthor@example.com]> * refactor: Improve accuracy calculation in perception test validation This commit refactors the `perceptiontest_val_aggregate_accuracy` function in the `utils.py` file. Instead of comparing the string representations of `answer_id` and `pred_id`, it now directly checks the `correct` field in the `accuracy` dictionary. This change ensures more accurate calculation of the overall accuracy in the perception test validation. Co-authored-by: [Co-author Name] <[coauthor@example.com]> * Refactor accuracy calculation in perception test validation * feat: Add SRT_API model to available models This commit adds the SRT_API model to the list of available models in the `__init__.py` file. This model can now be used for evaluation and testing purposes. Co-authored-by: [Co-author Name] <[coauthor@example.com]> --------- Co-authored-by: [Co-author Name] <[coauthor@example.com]>
kcz358
pushed a commit
that referenced
this pull request
Sep 5, 2024
* feat: Add multi-choice parsing and processing functions This commit adds two new functions to the `egoschema/utils.py` file: `get_multi_choice_info` and `parse_multi_choice_response`. These functions are used to parse and process multi-choice responses in the Egoschema task. The `get_multi_choice_info` function extracts information about the available choices from the input document, while the `parse_multi_choice_response` function parses the generated response and returns the predicted index. These functions are essential for accurately processing multi-choice answers in the Egoschema task. * feat: Add regex-based parsing for multi-choice predictions This commit enhances the `perceptiontest_val_process_results_mc` function in the `utils.py` file. It introduces regex-based parsing to extract the predicted choice from the raw text prediction. If a match is found for A, B, C, or D, the matched letter is used as the prediction. Otherwise, an empty string is set as the prediction. This improvement ensures accurate processing of multi-choice predictions in the perception test validation. Co-authored-by: [Co-author Name] <[coauthor@example.com]> * refactor: Improve accuracy calculation in perception test validation This commit refactors the `perceptiontest_val_aggregate_accuracy` function in the `utils.py` file. Instead of comparing the string representations of `answer_id` and `pred_id`, it now directly checks the `correct` field in the `accuracy` dictionary. This change ensures more accurate calculation of the overall accuracy in the perception test validation. Co-authored-by: [Co-author Name] <[coauthor@example.com]> * Refactor accuracy calculation in perception test validation * feat: Add SRT_API model to available models This commit adds the SRT_API model to the list of available models in the `__init__.py` file. This model can now be used for evaluation and testing purposes. Co-authored-by: [Co-author Name] <[coauthor@example.com]> --------- Co-authored-by: [Co-author Name] <[coauthor@example.com]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before you open a pull-request, please check if a similar issue already exists or has been closed before.
When you open a pull-request, please be sure to include the following
Thank you for your contributions!