[Backend] Support embedding model

[L-jasmine]

No description provided.

[L-jasmine]

@juntao @jmbejar

If the model-card repository contains an embedding.json file, the corresponding embedding model will be downloaded, and both the embedding and chat models will be loaded.

moxin-org/model-cards#8

[L-jasmine]

https://github.com/moxin-org/moxin/pull/242/files#diff-d406d942d2259b118601d9d68766930f7b8a6bf5d6700fb22011065649d4024fR207-R208

We can set the environment variable MODEL_CARDS_REPO at compile time to control which model-card repository the backend should clone.

[juntao]

Thanks. Please see my comment on the model card repo on the startup parameters for the nomic-embed model.

Also, when does the moxin app download the embedding model? What if the embedding model is not yet fully downloaded when the user starts to chat?

[jmbejar]

Thanks. Please see my comment on the model card repo on the startup parameters for the nomic-embed model.

Also, when does the moxin app download the embedding model? What if the embedding model is not yet fully downloaded when the user starts to chat?

The PR approach is to download the embedding model at Moxin's start time.. which takes many seconds in my machine, but it could be slower for other people.

Maybe it is a good start to move this download to a background task, having the chat interactions to use it whenever is available. Of course, we would need to show some UI indication to communicate to the user if the text embedding is in use or not, but we can do it in a separate PR.

[jmbejar]

@L-jasmine I tested this PR but the embedding json file was not downloading in my first attempts. Only when I removed the downloaded models-cards repo, the app recreated it (cloned it) and the embedding model was downloaded. So, I'm not sure if existing users will have the embedding models downloaded.

Looks like the "git pull" (in the models-cards repo folder) is not working for some reason.

[jmbejar]

@juntao @L-jasmine I'm not merging this because I feel the commented issues are worth to be addressed.

However, I'm curious to know what you think about. Do you think it is possible/convenient to download the embedding model on a thread to do not block application startup. Or we should just merge and try to address it on the frontend somehow?

BTW @L-jasmine let me know if you were able to reproduce the other issue, when you already have a local copy of the model-cards repo but fails to pull the new json file.

[juntao]

Yeah. Let's download it in a separate thread. We could prevent the user from chatting before this download completes. Thanks. @L-jasmine

[jmbejar]

@L-jasmine Tested again. Here are the results:

1- The git pull issue is resolved 👍

2- I'm hitting an issue with the download of the embedded schema in the background.
I tested having a chat conversation while the embedded was being downloaded, but later I realized I need to reload the load manually (from the UI) to have the embedded included in the chat section.

It may require some thinking from the UI to properly handle the first time the embedded is download, how the user is informed about. But let's handle it in another PR.

Accepting and merging now

[L-jasmine]

@L-jasmine Tested again. Here are the results:

1- The git pull issue is resolved 👍

2- I'm hitting an issue with the download of the embedded schema in the background. I tested having a chat conversation while the embedded was being downloaded, but later I realized I need to reload the load manually (from the UI) to have the embedded included in the chat section.

It may require some thinking from the UI to properly handle the first time the embedded is download, how the user is informed about. But let's handle it in another PR.

Accepting and merging now

Yes, because I have no way to notify the frontend that the embedding model has already been downloaded, so I can only load the embedding model during the reload.