Issue in ChatVertexAI Generation: Cannot read properties of undefined (reading 'text')

[jeloi]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

A very basic chain:

  const runnable = RunnableSequence.from<ChainInputs, DecomposeChainOutput>([
    {
      input: (i: ChainInputs) => i.input,
      chat_history: (i: ChainInputs) =>
        i.chat_history?.length
          ? formatChatHistoryForPrompt(i.chat_history)
          : [],
      customizationPrompt: () => {
        return fullConfigs.customizationPrompt;
      },
    },
    promptTemplate,
    model,
    new StringOutputParser(),
    }),
    ```
    
    Only potential relevant callout is using a custom safetyHandler:
    ```
        return new ChatVertexAI({
      modelName: modelParams.modelName,
      safetyHandler: {
        handle: (response) => {
          // eslint-disable-next-line @typescript-eslint/ban-ts-comment
          // @ts-ignore
          if (response.data?.candidates[0].finishReason === "SAFETY") {
            throw new InvalidRequestError(
              "Safety error - Gemini blocked response",
              response.data,
            );
          }
          return response;
        },
      },
      ...devAuthOptions,
      ...modelParams,
    });
    ```

### Error Message and Stack Trace (if applicable)

TypeError: Cannot read properties of undefined (reading 'text')
at ChatVertexAI.generate (file:///Users/user/code/projects/internal1/node_modules/.pnpm/@langchain+google-common@0.1.1@langchain+core@0.3.0_zod@3.23.5/node_modules/@langchain/google-common/dist/chat_models.js:275:64)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Promise.allSettled (index 0)
at async ChatVertexAI._generateUncached (file:///Users/user/code/projects/internal1/node_modules/.pnpm/@langchain+core@0.3.0_openai@4.40.0/node_modules/@langchain/core/dist/language_models/chat_models.js:182:29)
at async ChatVertexAI.invoke (file:///Users/user/code/projects/internal1/node_modules/.pnpm/@langchain+core@0.3.0_openai@4.40.0/node_modules/@langchain/core/dist/language_models/chat_models.js:56:24)
at async RunnableSequence.invoke (file:///Users/user/code/projects/internal1/node_modules/.pnpm/@langchain+core@0.3.0_openai@4.40.0/node_modules/@langchain/core/dist/runnables/base.js:1144:33)
at async RunnableSequence.invoke (file:///Users/user/code/projects/internal1/node_modules/.pnpm/@langchain+core@0.3.0_openai@4.40.0/node_modules/@langchain/core/dist/runnables/base.js:1150:27)
at async RunnableSequence.invoke (file:///Users/user/code/projects/internal1/node_modules/.pnpm/@langchain+core@0.3.0_openai@4.40.0/node_modules/@langchain/core/dist/runnables/base.js:1150:27)
at async forward (file:///Users/user/code/projects/internal1/node_modules/.pnpm/langsmith@0.1.41@langchain+core@0.3.0_langchain@0.3.5_openai@4.40.0/node_modules/langsmith/dist/evaluation/_runner.js:670:9)
at async RetryOperation._fn (/Users/user/code/projects/internal1/node_modules/.pnpm/p-retry@4.6.2/node_modules/p-retry/index.js:50:12)

### Description

Encountering an issue with ChatVertexAI where the above error is thrown.

This is after bumping to google-common 0.1.1.

Appears to be coming from this line: https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-google-common/src/chat_models.ts#L350

Unclear what the conditions are that cause this, it happens intermittently.

Context from Allen: If you want to hack it, I'd change it to ret?.generations?.[0]?.text ?? "" but I'll have a slightly better fix in (hopefully) a few days

cc @afirstenberg 

### System Info

    "@langchain/community": ">=0.3.0 <0.4.0",
    "@langchain/core": "^0.3.16",
    "@langchain/google-vertexai": "0.1.0",
    
    Node v21.7.3

[dosubot]

Hello, @jeloi! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to resolve this issue.

To resolve the "Cannot read properties of undefined (reading 'text')" error in ChatVertexAI when using LangChain.js with google-common 0.1.1, ensure that the candidateContent.parts array is properly checked for the presence of the text property before accessing it. The code should include checks for the presence of text in the relevant functions.

Here's a snippet that demonstrates how to handle this:

if (candidateContent?.parts.length === 1 && candidateContent.parts[0].text) {
  content = candidateContent.parts[0].text;
} else {
  content = candidateContent.parts.map((p) => {
    if ("text" in p) {
      return {
        type: "text",
        text: p.text,
      };
    } else if ("executableCode" in p) {
      return {
        type: "executableCode",
        executableCode: p.executableCode,
      };
    } else if ("codeExecutionResult" in p) {
      return {
        type: "codeExecutionResult",
        codeExecutionResult: p.codeExecutionResult,
      };
    }
    return p;
  });
}

Additionally, ensure that the responseToParts function safely extracts parts from the response data:

function responseToParts(response: GoogleLLMResponse): GeminiPart[] {
  const responseData = responseToGenerateContentResponseData(response);
  const parts = responseData?.candidates?.[0]?.content?.parts ?? [];
  return parts;
}

If the error persists, verify that the data structure being passed to these functions is correctly formatted and contains the expected properties ^[1].

To continue talking to Dosu, mention @dosu.

[afirstenberg]

@jacoblee93 - Please assign this to me, but I hope to have the fix in place as part of #6999

[jacoblee93]

Thank you @afirstenberg!