Expand gr.Chatbot() functionality + Refactor Tracking Issue

[dawoodkhan82]

TODOs

Additional Chatbot Features

• Chatbot does not show PDF file #4667
• Multimodal Input TextBox #4668
• chatbot buttons support #4763
• Allow a developer to add various Gradio components in user/bot responses
• Assign names and avatars in gr.Chatbot #3769
• Add names to chatbot
• Copy to clipboard button for messages
• Support for long messages (this might not be an issue. Need to double check)
• Option to disable markdown and/or latex
• Provide a few chatbot styles for devs to choose from. (Chatbot: limit width of chat bubbles to be the size of the messages #4420)
• Add liking/voting functionality to gr.Chatbot messages #5250
• Add intermediate steps (Display intermediate work (e.g. LLM agent thought process) inside Chatbot messages  #5346)

[dawoodkhan82]

To simplify the gr.Chatbot() api, we can provide a ChatMessageHistory class (similar to LangChain). The class manages the chatbot history and includes methods such as add_user_message, add_bot_message, get_last_message, etc. Right now, the data structure passed into the chatbot is tuples inside a list of lists, which can get very confusing. ChatMessageHistory is how LangChain manages chat history (https://docs.langchain.com/docs/components/memory/chat_message_history), so it will be familiar with devs using LangChain. Thoughts @abidlabs?

[abidlabs]

I do see value in ChatMessageHistory, but I worry that it comes with a not-so-insignificant cost, which is creating a layer of abstraction over a relatively simple data structure (nested lists), which the vast majority of Python developers know and understand. This causes confusion later on when people want to do basic things that our class does not support -- this is a problem people run into frequently with langchain and I don't think we should follow that path

[dawoodkhan82]

I see what you mean, but the data structure can get messy fairly quickly. Especially when its tuples nested within lists of lists, and when files get involved. Also we have heard feedback, that the chatbot API is confusing, so I think something needs to be done to simplify it. Of course having the ChatInterface abstraction will help too. Having a ChatMessageHistory class can also be helpful when we want to extend the functionality of the chatbot, without having to worry about breaking changes of the underlying data structure. The criticisms that I see of LangChain are because of the fact that added too. many abstraction layers and pointless helper functions. As long as we make sure we don't over engineer the ChatMessageHistory, that should be fine, no?

[abidlabs]

Of course having the ChatInterface abstraction will help too. Having a ChatMessageHistory class can also be helpful when we want to extend the functionality of the chatbot, without having to worry about breaking changes of the underlying data structure

We'll need to be backwards compatible anyways since many users will continue to pass lists of data so I don't think this will get us out of it. Plus if someone is accessing the Chatbot via the Clients, they'll still get and pass data as lists of data, so again we'll need to fully support lists of data anyways.

Also we have heard feedback, that the chatbot API is confusing, so I think something needs to be done to simplify it.

The feedback I've seen (e.g. #3510) is that we need a simpler way to create a fully functioning Chatbot demo, not that handling the messages is complicated. I agree that nested lists can be slightly confusing, but this class makes things more verbose in my opinion. Here are some examples:

Function	Current	with the abstraction
Initializing	messages = []	messages = ChatMessageHistory()
Add user and bot message	messages.append([user_message, bot_message])	messages.add_user_message(user_message) messages.add_bot_message(bot_message)
Get last message	messages.pop()[-1]	messages.get_last_message()

Imo, the value of the abstraction is just not there, and now a person has to look up the relevant functions in the documentation every time they need to do something new (e.g. how do I clear the message history) instead of building on their existing knowledge of python

[dawoodkhan82]

I see, yeah didn't consider the gradio client, plus the fact that we would still want the chatbot to be backwards compatible.

Function	Current	with the abstraction
Initializing	messages = []	messages = ChatMessageHistory()
Add user and bot message	messages.append([user_message, bot_message])	messages.add_user_message(user_message) messages.add_bot_message(bot_message)
Get last message	messages.pop()[-1]	messages.get_last_message()

Also, makes sense that the abstraction isn't very useful for the simple functions listed above.

Thanks for talking this thru!

[cpwan]

How is the progress on "Allow a developer to add various Gradio components in user/bot responses"?

I have been trying to hack the chatInterface, chatbot to include additional output components, e.g. a Textbox for suggested questions, which could be outside of the chatbot block in the UI. The closest I can get is to write a class for nested component so that I can do sth like ChatInterface(fn=new_fn, chatbot=NestedComponents(components=[gr.Chatbot(), gr.Textbox()])) and the new_fn takes the same arguments message, history and return the modified history, but the history is now a tuple of values for multiple output components.

But error occurs when ChatInterface initializes event listeners for the cancel buttons. Any suggestions?

[edwinjosechittilappilly]

I would also appreciate if I could add various gradio component as output.
Example I am making a RAG Chat bot and these components below he bot response could be used to show the sources or just the source title. Just like what we see in bing chat as sources.

[abidlabs]

I've added a Chatbot label instead (more appropriate as new issues are being created), I think we can close this tracking issue. Feel free to reopen if you feel differently @dawoodkhan82