LLM Tool for open-source models on Hugging Face Hub?

[faaany]

At the moment, the only way to use open-source models is to choose models from the Azure Machine Learning model catalog and create an endpoint for it. What if I want to use open-source models from Hugging Face Hub, can we have a LLM Tool that support Hugging Face models?

[0mza987]

@faaany Could you please elaborate on your user scenario? By saying "use open-source models from Hugging Face Hub", do you mean you want to host the models on places other than azure endpoint and do the inference, or download the models to local? And how would you like the LLM tool to be used, could you please provide a code snippet to describe your expectations to the LLM tool?

[faaany]

Hi @0mza987 , thanks for replying! I was asking for whether there is a way for me to use open-source models hosted on HuggingFace Hub, e.g. https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1.

I later figured it out that I could use TGI and pass the model endpoint to promptflow, so we can close this issue. Thanks again!