-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PR/image analysis #323
Merged
art-from-the-machine
merged 24 commits into
art-from-the-machine:main
from
YetAnotherModder:PR/Image_analysis-http-gui
Jan 7, 2025
Merged
PR/image analysis #323
art-from-the-machine
merged 24 commits into
art-from-the-machine:main
from
YetAnotherModder:PR/Image_analysis-http-gui
Jan 7, 2025
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add additional IMAGE_GPT_SECRET_KEY.txt for separate LLM access Add new Image_Manager.py to handle image generation, management and removal after use. Add new image LLM config values to config loader Add new image LLM definitions file Add new image LLM prompt definitions Add new config values definitions for image LLM functionality Modify conversation.py to generate and image_manager upon startup and use it to handle image analysis calls. Modify gameable, Skyrim and Fallout4.py to add a get_image_filepath() Add image_client_instance creation to mantella_route.py Add new functions to message_thread.py to delete, replace and add specific message types. Add new message type handling for image_message & image_description_message. Add two new classes to messages.py : image_message & image_description_message Modified openai_client.py to add a new class image_client. Add new method : streaming_one_message_call. Refactored the openai_client class to avoid redundancies when creating the subclass. Modified output_manager.py with three new methods generate_simple_response(), process_simple_response(), run_async() & get_image_filepath() to allow iterative image generation
-Fix __config initialization in Skyrim.py -Fix operator use in image_manager.py
Misc fixes
- Improve Image LLM definitions - Refactor messages.py and image_manager.py to make image_manager.py in charge of encoding. -Add functionality for image resizing according to in game variables to image_manager.py -Made the error messages a bit less obnoxious -Refactor method process_image_analysis() so it doesn't crash in case of absent images and add error messages. -Change find_most_recent_jpg() so that it excludes files ending in vr and also that it can't return the same filepath more than once in the same conversation by comparing to the deletion array. -Change delete_images_from_file() so that it checks for images ending with _vr as well for a more complete cleanup.
Updated image prompt definitions and manager to take into account the {game} variable
Add additional instructional comments
Add pillow as a requirement (for image resizing)
New config option to allow the user to select if the Steam screenshots are deleted after Mantella use or not Update definition for Skyrim to indicate the two possible filepaths (SUP_SKSE or Steam) New methods in imagemanager to allow to process hints and manage variable delay for Steam screenshot analysis. Minor bugfix to attempt_to_add_most_recent_image_to_deletion_array () to avoid errors when self.KEY_CONTEXT_CUSTOMVALUES_VISION_READY is empty. Refactored create_message_from_image() to account for non-VR Steam screenshots. Refactored process_image_analysis() to add hints support
gitignore : Updated to account for other secret keys communication_constants.py : Updated to allow compatibility with recent versions of Mantella image_llm_definitions.py : Changed definition and default value for FO4 VR since screenshots are now taken in-engine image_manager.py : attempt_to_add_most_recent_image_to_deletion_array() modified to account for Steam screenshot being enable instead of only VR being enabled due to FO4VR having in-engine screenshot capabilities resize_image() modified to take into account images with alpha channels (e.g. png) image_to_base64() modified to convert RGBA to RBG create_message_from_image() modified to take into account the customcontext variable KEY_CONTEXT_CUSTOMVALUES_VISION_ISUSINGSTEAMSCREENSHOT instead of relying on VR to select if the Steam screenshot folder will be parsed with : find_most_recent_jpg(). The function is also modified to look for png files instead of just jpgs (due to limitation with the in-engine screenshots of Skyrim)
art-from-the-machine
approved these changes
Jan 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes multiple changes to Mantella to allow image analysis.
The aim was to allow two things :
Key points:
The new image manager.py handles multiple tasks:
Potential future upgrades :