Support MongoDB for custom data layer

[sandangel]

• fix: correct typing for _data_layer
• draft implement for mongodb data layer

[willydouhard]

@tpatel #796 #796

[tpatel]

@sandangel Thanks for your contribution, this is a great idea!

I have some feedback:

• The best way to implement a custom data layer should be to only provide a class that overrides BaseDataLayer (this is what you've done with MongoDataLayer)
• There should be no changes to the rest of the chainlit code (outside of exposing this class in the package). Users are expected to do cl_data._data_layer = MongoDataLayer() in their chainlit app, to configure the data layer.

Let me know if you have any question!

[sandangel]

Sure, let me revert the change in config. I will update the PR tomorrow.

[sandangel]

@tpatel , I have a small PR to fix typing for the data layer first to help me test my change on local. Could you please check them first? #802

Once I finished integration tests I will update this PR.

[hayescode]

@sandangel thank you for making this! This is a great step in the right direction!

One question: Can the blob storage be split from the class to give users the ability to use MongoDB in other cloud environments like Azure/Google?

[sandangel]

@hayescode
For now I'm changing the approach to mimic the literalai client instead. Ideally users can implement their own upload function that use different blob storage

[sandangel]

hi @tpatel , I finished the mongodb custom data layer on my local and now testing. However I could not enable the chat history. Do you have any idea?

[sandangel]

Oh I found the issue. I need to use cl_data._data_layer instead.

[Jimmy-Newtron]

This is an excellent work 👍

I am really enthusiast of this and it would be great if Elasticsearch can be used as well

You can use Elasticsearch as a vector store for RAG Assistants

[sandangel]

@Jimmy-Newtron If you look at the mongo_api.py, you'll see I'm using mongo client directly. We might need to have a database adapter to support different databases. Perhaps you can use elastic search for vector search and mongo db for storing chat data. Others feel free to have a look at how I handle mongo_api client to mimic literalai client.

Implementing the BaseDataLayer has been very confusing because of the mapping between chainlit types and literalai types. However, since chainlit data layer only use Literal Client to interact with Literal AI, we can implement LiteralAI Client for Mongo DB instead and let chainlit handle the type mapping between chainlit and literalai.
I think it's cleaner than trying to implement the BaseDataLayer with both chainlit + literalai

[tpatel]

I'm curious to learn more about what is confusing with the types? My goal is to make sure it's smooth to implement the data layers.

One thing that I've noticed in your PR is that you are inheriting a class from ChainlitDataLayer, which shouldn't be needed. The ideal scenario is to use ChainlitDataLayer only as an exemple and to inherit from BaseDataLayer.

[sandangel]

hi @tpatel . Thanks for checking. In my cases, I feel confused because lots of data types were imported from literalai and then mapped to chainlit data types. In chainlit data types, there is also a confusion for having multiple types to describe the same object. Like User/UserDict. Ideally, all the DSL should be defined in chainlit, and literalai will just import those instead. Also in base datta layer, all the methods are not clear and well designed to be implemented properly. The LiteralAI client has a better interface to implement and really straight forward.

[sandangel]

Closing, since our team decided to use postgres instead