Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wip(feat/paste): images dropzone #161

Merged
merged 23 commits into from
Nov 6, 2024
Merged

Conversation

MarcMcIntosh
Copy link
Collaborator

@MarcMcIntosh MarcMcIntosh commented Oct 21, 2024

Add images to chat

Description

User can add images to chat by either clicking a button or drag and drop.
Reties also work

Allow the suer to drop images into chat.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactoring (no functional changes, only code improvements)
  • Documentation update

How to Test

  1. run the lsp with BYOK
  2. run chat
  3. Click the image icon
  4. add files
  5. ask question about the images
  6. send chat

repeat for drag and drop, but also include non-image files

Screenshots (if applicable)

Checklist

  • My code follows the code style of this project.
  • I have performed a self-review of my code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • Any dependent changes have been merged and published in downstream modules.
  • I have updated the documentation where necessary.

Linked Issues

Additional Notes

Merge once it works without needing byok

@MarcMcIntosh MarcMcIntosh changed the base branch from main to alpha October 22, 2024 16:35
 Conflicts:
	src/app/middleware.ts
	src/components/Chat/Chat.tsx

 On branch feat/paste-images-dropzone
 Your branch is up to date with 'origin/feat/paste-images-dropzone'.
 All conflicts fixed but you are still merging.
 Changes to be committed:
	modified:   package-lock.json
	modified:   package.json
	modified:   src/app/middleware.ts
	modified:   src/components/Chat/Chat.tsx
	modified:   src/components/ChatContent/AssistantInput.tsx
	modified:   src/components/ComboBox/ComboBox.tsx
	modified:   src/components/Markdown/Markdown.module.css
	modified:   src/components/Markdown/Markdown.tsx
	modified:   src/events/index.ts
	modified:   src/features/Chat/Chat.tsx
	modified:   src/features/History/historySlice.ts
	modified:   src/features/ThreadHistory/ThreadHistory.tsx
	modified:   src/hooks/useEventBusForIDE.ts
	modified:   src/hooks/usePatchActions.ts
	modified:   src/services/refact/prompts.ts
	modified:   src/utils/copyChatHistoryToClipboard.ts
	new file:   src/utils/fallbackCopying.ts
@MarcMcIntosh MarcMcIntosh marked this pull request as ready for review October 25, 2024 13:12
const err = file.errors.reduce<string>((acc, cur) => {
return acc + `${cur.code} ${cur.message}\n`;
}, "");
return `could not atttach ${file.file.name}: ${err}`;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small typo here
image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where chatGenerateTitleThunk, you are sending firstUserMessage, but its content is no longer a string, it's an object of objects with images attached and actual user query, so handling of chat title should be also modified
image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or other workaround could be to transform firstUserMessage within historyListener in that way, that it's structure will become as old-structured message and will be sent to chatGenerateTitleThunk

Possible workaround

interface InputContent {
    m_type: string;
    m_content: string;
}

interface InputObject {
    role: string;
    content: InputContent[];
}

interface OutputObject {
    role: string;
    content: string;
}

function transformObject(input: InputObject): OutputObject {
    // Extract the m_content of the first object in the content array
    const firstContent = input.content[0].m_content;

    // Construct the new object with in old-structured way
    const output: OutputObject = {
        role: input.role,
        content: firstContent
    };

    return output;
}

@@ -264,10 +265,12 @@ export const ChatForm: React.FC<ChatFormProps> = ({
size="1"
type="submit"
/>
{/** TODO: disabled prop, title and size props */}
<AttachFileButton />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AttachFileButton should be placed before PaperPlaneButton, on the left side

Copy link
Contributor

@alashchev17 alashchev17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments are left above

Copy link
Contributor

@alashchev17 alashchev17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏻

Base automatically changed from alpha to main October 29, 2024 15:49
@MarcMcIntosh MarcMcIntosh changed the base branch from main to alpha October 29, 2024 17:21
@MarcMcIntosh MarcMcIntosh merged commit 0eb4b8b into alpha Nov 6, 2024
2 checks passed
@MarcMcIntosh MarcMcIntosh deleted the feat/paste-images-dropzone branch November 6, 2024 13:48
olegklimov pushed a commit that referenced this pull request Nov 7, 2024
* WIP(upload files): add a file drag and drop handler for chat.

* wip(images): add some ui components for interaction.

* wip(images): add attach button.

* wip(ui attached images): add a simple ui to see attached files.

* refactor(attached images): manage attached images in redux.

* wip(images): add images to the user content.

* wip(images): send images using kyok and an upstream branch of the lsp `multimodality_plus`

* wip(image user messages): render only a string message for now.

* wip(images): rendering user images in chat

* wip(images): render multiple user images in chat.

* wip(images): refactor to reuse the original component for the component is an array.

* wip(image): add retry functionality

* ui(retry): add buttons for adding and removing images when retrying a question.

* ui(images): add a better button for removing the image

* feat(images): add error messages for unsupported files.

* feat(images): add error and warning handlers for adding files.

* refactor(api reset): reset ping first.

* typo(dropzone): atttach

* ui(image button): change the order for send and attach buttons.

* fix(BYOK): title generation response won't have `"metering_balance` when using BYOK.

* refactor(images): place images before the user message
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants