feat: Vision Support + New UI #1203

danny-avila · 2023-11-20T14:28:17Z

Summary

Notes:

This is not considered stable until I cut a release (v0.6.2)
This is a breaking change: the main route has changed to /c/new from /chat/new
As of now, only image attachments are supported. URLs will be supported within a few days
Images are resized according to the dimensions the gpt-4-vision expects
- Since the only storage strategy is local right now, numerous safeguards have been implemented, including optimizing image size
If you would like to toggle this feature off, this is not ready yet, but will be in the next few days
- It might be worth disabling until SafeSearch or remote upload services are integrated to protect from unwanted material
- Also if you are mindful of your server's hard disk space (though the resizing is really optimal for image quality/space)
OpenAI Endpoint only. Vision support for Plugins may come after the holiday.
All images are handled locally--remote services like Firebase and S3 are planned, and I would like contributions for this.
Investigating: minor but docker may or may not load images when the image request is first sent (after first attaching them to the message).
- This is due to how docker handles internal networking and will find a solution
Configurable file limits

Limits:

Only 10 images per request and 20 MB per file max (enforced by API)
25 MB total per request

The next big priority for LibreChat will be custom GPTs/Assistants support

Change Type

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Testing

Tests will fail until they are updated

Checklist

My code adheres to this project's style guidelines
I have performed a self-review of my own code
I have commented in any complex areas of my code
I have made pertinent documentation changes
My changes do not introduce new warnings
I have written tests demonstrating that my changes are effective or that my feature works
Local unit tests pass with my changes
Any changes dependent on mine have been merged and published in downstream modules.

…ypes in TS

…ategize new plan to separate react dependent packages

…acks in a way that keeps them during component unmount, initial delete handling

…r fidelity

…ient, made note to fix for PluginsClient

…ildMessages method, count tokens for nested objects/arrays

…lities

… role and image_urls passed

…ctly set and remove TTL

TODO: file size, type, amount validations, making sure they are styled right, and making sure you can add images from the clipboard/dragging

… Presentation component so the useDragHelpers hook has ChatContext

…ier re-use

…, pass more file metadata in messages

… in lightmode, make menu more ux friendly

…ll fail until addressed

…h, to fix temp deletion

…ead of `ChatGPT`

* feat: add timer duration to showToast, show toast for preset selection * refactor: replace old /chat/ route with /c/. e2e tests will fail here * refactor: move typedefs to root of /api/ and add a few to assistant types in TS * refactor: reorganize data-provider imports, fix dependency cycle, strategize new plan to separate react dependent packages * feat: add dataService for uploading images * feat(data-provider): add mutation keys * feat: file resizing and upload * WIP: initial API image handling * fix: catch JSON.parse of localStorage tools * chore: experimental: use module-alias for absolute imports * refactor: change temp_file_id strategy * fix: updating files state by using Map and defining react query callbacks in a way that keeps them during component unmount, initial delete handling * feat: properly handle file deletion * refactor: unexpose complete filepath and resize from server for higher fidelity * fix: make sure resized height, width is saved, catch bad requests * refactor: use absolute imports * fix: prevent setOptions from being called more than once for OpenAIClient, made note to fix for PluginsClient * refactor: import supportsFiles and models vars from schemas * fix: correctly replace temp file id * refactor(BaseClient): use absolute imports, pass message 'opts' to buildMessages method, count tokens for nested objects/arrays * feat: add validateVisionModel to determine if model has vision capabilities * chore(checkBalance): update jsdoc * feat: formatVisionMessage: change message content format dependent on role and image_urls passed * refactor: add usage to File schema, make create and updateFile, correctly set and remove TTL * feat: working vision support TODO: file size, type, amount validations, making sure they are styled right, and making sure you can add images from the clipboard/dragging * feat: clipboard support for uploading images * feat: handle files on drop to screen, refactor top level view code to Presentation component so the useDragHelpers hook has ChatContext * fix(Images): replace uploaded images in place * feat: add filepath validation to protect sensitive files * fix: ensure correct file_ids are push and not the Map key values * fix(ToastContext): type issue * feat: add basic file validation * fix(useDragHelpers): correct context issue with `files` dependency * refactor: consolidate setErrors logic to setError * feat: add dialog Image overlay on image click * fix: close endpoints menu on click * chore: set detail to auto, make note for configuration * fix: react warning (button desc. of button) * refactor: optimize filepath handling, pass file_ids to images for easier re-use * refactor: optimize image file handling, allow re-using files in regen, pass more file metadata in messages * feat: lazy loading images including use of upload preview * fix: SetKeyDialog closing, stopPropagation on Dialog content click * style(EndpointMenuItem): tighten up the style, fix dark theme showing in lightmode, make menu more ux friendly * style: change maxheight of all settings textareas to 138px from 300px * style: better styling for textarea and enclosing buttons * refactor(PresetItems): swap back edit and delete icons * feat: make textarea placeholder dynamic to endpoint * style: show user hover buttons only on hover when message is streaming * fix: ordered list not going past 9, fix css * feat: add User/AI labels; style: hide loading spinner * feat: add back custom footer, change original footer text * feat: dynamic landing icons based on endpoint * chore: comment out assistants route * fix: autoScroll to newest on /c/ view * fix: Export Conversation on new UI * style: match message style of official more closely * ci: fix api jest unit tests, comment out e2e tests for now as they will fail until addressed * feat: more file validation and use blob in preview field, not filepath, to fix temp deletion * feat: filefilter for multer * feat: better AI labels based on custom name, model, and endpoint instead of `ChatGPT`

danny-avila added 30 commits November 18, 2023 16:47

feat: add timer duration to showToast, show toast for preset selection

5f59d72

refactor: replace old /chat/ route with /c/. e2e tests will fail here

658986d

refactor: move typedefs to root of /api/ and add a few to assistant t…

3c0d60f

…ypes in TS

refactor: reorganize data-provider imports, fix dependency cycle, str…

6544d0e

…ategize new plan to separate react dependent packages

feat: add dataService for uploading images

a5ba6f2

feat(data-provider): add mutation keys

5792696

feat: file resizing and upload

f8679cc

WIP: initial API image handling

3329b1f

fix: catch JSON.parse of localStorage tools

0482408

chore: experimental: use module-alias for absolute imports

769b913

refactor: change temp_file_id strategy

124a5c9

fix: updating files state by using Map and defining react query callb…

087acd4

…acks in a way that keeps them during component unmount, initial delete handling

feat: properly handle file deletion

a12506e

refactor: unexpose complete filepath and resize from server for highe…

68ee9c3

…r fidelity

fix: make sure resized height, width is saved, catch bad requests

9257860

refactor: use absolute imports

5d9cb01

fix: prevent setOptions from being called more than once for OpenAICl…

47e9d6f

…ient, made note to fix for PluginsClient

refactor: import supportsFiles and models vars from schemas

9020d72

fix: correctly replace temp file id

7524c26

refactor(BaseClient): use absolute imports, pass message 'opts' to bu…

10ed007

…ildMessages method, count tokens for nested objects/arrays

feat: add validateVisionModel to determine if model has vision capabi…

dd6b434

…lities

chore(checkBalance): update jsdoc

99b54a0

feat: formatVisionMessage: change message content format dependent on…

1c98d9a

… role and image_urls passed

refactor: add usage to File schema, make create and updateFile, corre…

740e8de

…ctly set and remove TTL

feat: working vision support

03af055

TODO: file size, type, amount validations, making sure they are styled right, and making sure you can add images from the clipboard/dragging

feat: clipboard support for uploading images

e446450

feat: handle files on drop to screen, refactor top level view code to…

6973f74

… Presentation component so the useDragHelpers hook has ChatContext

fix(Images): replace uploaded images in place

475ed60

feat: add filepath validation to protect sensitive files

639f8f5

fix: ensure correct file_ids are push and not the Map key values

3858145

danny-avila added 3 commits November 19, 2023 19:07

fix: react warning (button desc. of button)

c42b19d

refactor: optimize filepath handling, pass file_ids to images for eas…

f270065

…ier re-use

refactor: optimize image file handling, allow re-using files in regen…

6032133

…, pass more file metadata in messages

danny-avila marked this pull request as draft November 20, 2023 15:21

danny-avila added 20 commits November 20, 2023 14:15

feat: lazy loading images including use of upload preview

114e2b5

fix: SetKeyDialog closing, stopPropagation on Dialog content click

bc9b3ae

style(EndpointMenuItem): tighten up the style, fix dark theme showing…

83800d9

… in lightmode, make menu more ux friendly

style: change maxheight of all settings textareas to 138px from 300px

8d6f8a5

style: better styling for textarea and enclosing buttons

d623114

refactor(PresetItems): swap back edit and delete icons

724cb74

feat: make textarea placeholder dynamic to endpoint

2d4d0c2

style: show user hover buttons only on hover when message is streaming

28c10ee

fix: ordered list not going past 9, fix css

e09f3b2

feat: add User/AI labels; style: hide loading spinner

b1dfde9

feat: add back custom footer, change original footer text

c8c4ab6

feat: dynamic landing icons based on endpoint

3c08ad3

chore: comment out assistants route

d427b94

fix: autoScroll to newest on /c/ view

a63ca65

fix: Export Conversation on new UI

a763990

style: match message style of official more closely

6c05418

ci: fix api jest unit tests, comment out e2e tests for now as they wi…

aa20c2c

…ll fail until addressed

feat: more file validation and use blob in preview field, not filepat…

0ce7621

…h, to fix temp deletion

feat: filefilter for multer

ca4c8f0

feat: better AI labels based on custom name, model, and endpoint inst…

44d602b

…ead of `ChatGPT`

danny-avila marked this pull request as ready for review November 21, 2023 23:54

danny-avila changed the title ~~feat: Vision Support~~ feat: Vision Support + New UI Nov 22, 2023

danny-avila merged commit 317cdd3 into main Nov 22, 2023
3 checks passed

danny-avila deleted the vision branch November 22, 2023 01:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Vision Support + New UI #1203

feat: Vision Support + New UI #1203

danny-avila commented Nov 20, 2023 •

edited

Loading

feat: Vision Support + New UI #1203

feat: Vision Support + New UI #1203

Conversation

danny-avila commented Nov 20, 2023 • edited Loading

Summary

Change Type

Testing

Checklist

danny-avila commented Nov 20, 2023 •

edited

Loading