Add support for Vertex AI Model Garden / Custom Models

[afirstenberg]

See langchain-ai/langchain#8850 and langchain-ai/langchain#9367 and will talk to @lkuligin about their implementation.

The general idea will be to provide a standard way to either access instances of these models at a designated endpoint directly, or create a subclass that can do so. (The request and response format have a standard, but it is a vague standard in some places.)

[jacoblee93]

Would be great - I think this would likely look similar to SageMaker with an adapter class for different models?

https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/llms/sagemaker_endpoint.ts#L35

[afirstenberg]

Yup, that's along the lines of what I was thinking (possibly with some specific subclasses for popular or well-advertised models).

[dosubot]

Hi, @afirstenberg,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. From what I understand, the issue was opened to add support for Vertex AI Model Garden and Custom Models, with discussions around establishing a standard method for accessing these models at a designated endpoint or creating a subclass to do so. The request and response format have a standard, but there are some vague areas. You and @jacoblee93 have discussed potential approaches, including using an adapter class for different models and specific subclasses for popular or well-advertised models.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

[afirstenberg]

This should be left open. The issue is still valid and we'll be evaluating the state of things once we see what changes Gemini brings.

[dosubot]

@jacoblee93, could you please assist with evaluating the state of the issue related to adding support for Vertex AI Model Garden and Custom Models in langchainjs? The user has indicated that the issue is still valid and will be re-evaluated after the changes brought by Gemini. Thank you!

[AInoob]

Hi @afirstenberg, Gemini API is out, any estimate on when langchainjs will support Gemini?

[afirstenberg]

Hi @afirstenberg, Gemini API is out, any estimate on when langchainjs will support Gemini?

#3621 Added integration for the Google AI Studio version of Gemini and that will be in the next release. (Thanks to @alx13 for that work!)

We're currently discussing the best way to implement it for Vertex AI. If I'm working on it, It'll be this weekend.

[dosubot]

Hi, @afirstenberg,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. From what I understand, the issue you opened requests support for Vertex AI Model Garden and Custom Models in the langchainjs repository. Discussions have been ongoing about establishing a standard method for accessing these models at a designated endpoint or creating a subclass to do so. The issue is still relevant, and the team plans to evaluate it after the changes brought by Gemini. Additionally, integration for the Google AI Studio version of Gemini has been added, and discussions are ongoing for implementing it for Vertex AI.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

[afirstenberg]

This is still active - specifically targeting the models in Vertex AI Model Garden.

[Storkki]

@jacoblee93 Is there some news about how to use ChatVertexAI (or maybe VertexAI) for models on custom endpoint or for models that has other publisher than 'google'

[afirstenberg]

The most accurate news I can share at the moment is "I'm actively working on it". {:
Hoping to have a branch for folks to at least start looking at by the end of the week.

You specified two requirements here - exactly which models are you looking to use, those available with the "Model-as-a-Service" that Vertex is offering for Claude and Llama and some others? Or those from your model registry that you've deployed to your own endpoint?

[Storkki]

To be clearly, i need to use models that are available in my gcp project (using endpoints or publisher url). For example, i want to use LLaMa (which is on endpointId) and Claude (claude-3-5-sonnet@20240620) in Langchain VertexAI.

Endpoints that i use for api calls:

`projects/${projectId}/locations/${location}/endpoints/${endpointId}`
`projects/${projectId}/locations/${location}/publishers/${publisher}/models/${model}`;

[afirstenberg]

Got it. Thanks.
I'm focusing on the projects.locations.publishers.models first and then projects.locations.endpoints.
My goal, however, is to make it relatively easy to add more once some are in place.

Still hoping to be able to show something by end of week.

[Storkki]

Thank you! Good luck)

[Storkki]

@afirstenberg Hi, are there any updates?

[afirstenberg]

@Storkki - This is still very much work in progress, but you can see roughly where this will be going with #6999

Hopefully I can get anthropic, at least, completed and tested in short order and we can get this merged. And then move on to the next model to support. {:

(Sorry this has taken so long. Every time I got ready to commit, I thought I really should add one more thing. There are still many many more things.)

[Storkki]

@afirstenberg Thank you for your fantastic work!

Maybe you know Is there any plan to support self-hosted models through endpoints/modelEndpointId?

[afirstenberg]

@Storkki - That is the plan! The changes put in place for this should make Vertex endpoints easier to support. I'll be opening up new issues to track things individually.