vertexai: Allow json_mode in with_structured_output

[jzaldi]

PR Description

Adds a keyword parameter method='json_mode' to with_structured_output in ChatVertexAI to use controlled generation rather than function calling.

Relevant issues

Fixes #531, #503, #345, #144

Type

🆕 New Feature

Example

from pydantic import BaseModel
from langchain_google_vertexai import ChatVertexAI

class Response(BaseModel):
    response: str
    reason: str

model = (
    ChatVertexAI(model_name="gemini-1.5-pro-001")
    .with_structured_output(Response, method="json_mode")
)

model.invoke("What is the color of the sea?")

>> Response(response="The sea doesn't have a single color. It often appears blue due to the reflection of the sky, but it can also be green, turquoise, or even brown depending on depth, sediment, and the presence of microorganisms.", reason='The color of the sea is determined by the way it absorbs and reflects light, not an inherent pigment.')

[jzaldi]

We have an issue with nested pydantic models. The schema generated contains relative references and gemini api doesn't support that. I can implement a postprocessing function to get rid of refs in the meantime but is not ideal. @lkuligin wdyt?

[jzaldi]

Also I don't see any conflicts with what is being implemented in #469

[jzaldi]

Now we are only missing self referencing objects, something like this doesn't work and I don't think we can do anything about it if gemini's API doesn't support references:

class Status(BaseModel):
    substatus: "Status"

[alexminza]

This should also be added to ChatGoogleGenerativeAI

Reference: https://ai.google.dev/gemini-api/docs/structured-output