GH-4596: Handle candidates containing both text and tool calls in VertexAiGeminiChatModel

[q-nathangrand]

See: #4596

[q-nathangrand]

It didn't make sense to me to be turning one candidate into multiple generations, but perhaps I'm misunderstanding?

Previous behaviour collected all parts containing function tools and combined into one AssistantMessage; idea here was to take the same approach here if multiple parts contain text.

[q-nathangrand]

Just testing this further manually

[q-nathangrand]

Just testing this further manually

Seems to be working well

[ericbottard]

Hi @q-nathangrand, thanks for taking the time to submit a PR for this issue.

Could you provide a test case that exhibits the initial problem (ie a response that contains a mixture of FunctionCall and non-FunctionCall parts). Either as maybe a prompt that triggers such a response, or even better as an integration test.

Also, could you confirm that this issue could arise in both the streaming and non-streaming cases?

[q-nathangrand]

Hi @q-nathangrand, thanks for taking the time to submit a PR for this issue.

Could you provide a test case that exhibits the initial problem (ie a response that contains a mixture of FunctionCall and non-FunctionCall parts). Either as maybe a prompt that triggers such a response, or even better as an integration test.

Also, could you confirm that this issue could arise in both the streaming and non-streaming cases?

Hey,

I've only been testing it in the non-streaming version, but the responseCandidateToGeneration is used by both streaming and non-streaming code paths.

Integration test wise, not sure if I'll get time this week.

But the general idea is to encourage gemini to explain why it it calling tools as it does.

Here's a snippet (sorry for the scala/java mix)

import com.google.cloud.vertexai.VertexAI
import org.springframework.ai.chat.messages.{SystemMessage, UserMessage}
import org.springframework.ai.chat.prompt.Prompt
import org.springframework.ai.model.tool.DefaultToolCallingManager
import org.springframework.ai.support.ToolCallbacks
import org.springframework.ai.vertexai.gemini.schema.VertexToolCallingManager
import org.springframework.ai.vertexai.gemini.{VertexAiGeminiChatModel, VertexAiGeminiChatOptions}

import scala.collection.JavaConverters._

object Test extends App {

  private val vertexAi = new VertexAI.Builder()
    .setProjectId("your-project-id")
    .setLocation("us-east5")
    .build()

  private val toolCallbacks = ToolCallbacks.from(new TestTools())
  private val toolNames = toolCallbacks.map(_.getToolDefinition.name()).toSet.asJava

  private val options = VertexAiGeminiChatOptions
    .builder()
    .internalToolExecutionEnabled(true)
    .temperature(0)
    .model("gemini-2.5-pro")
    .toolCallbacks(toolCallbacks :_*)
    .toolNames(toolNames)
    .build()

  private val model = VertexAiGeminiChatModel
    .builder()
    .vertexAI(vertexAi)
    .defaultOptions(options)
    .toolCallingManager(new VertexToolCallingManager(DefaultToolCallingManager.builder().build()))
    .build()

  private val response = model.call(
    Prompt
      .builder()
      .messages(
        SystemMessage.builder().text(
          "You MUST include reasoning when you issue tool calls."
        ).build,
        UserMessage.builder().text("Set an alarm for an hour from now, and tell me what time that was for").build
      )
      .build()
  )

  println(response.getResult.getOutput.getText)

}

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.context.i18n.LocaleContextHolder;

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;

public class TestTools {

    @Tool(description = "Get the current date and time in the user's timezone")
    String getCurrentDateTime() {
        System.out.println("----- Tool call: getCurrentDateTime begin -----");
        String localDateTime = LocalDateTime.now().atZone(LocaleContextHolder.getTimeZone().toZoneId()).toString();
        System.out.println("----- Tool call: getCurrentDateTime end -----");
        return localDateTime;
    }

    @Tool(description = "Set a user alarm for the given time, provided in ISO-8601 format")
    void setAlarm(String time) {
        System.out.println("----- Tool call: setAlarm being -----");
        LocalDateTime alarmTime = LocalDateTime.parse(time, DateTimeFormatter.ISO_DATE_TIME);
        System.out.println("----- Tool call: setAlarm end -----");
    }

}

Here's a couple of grabs from debugger showing the issue

[ericbottard]

Thanks for the extensive response, that should help us reproduce the issue and create an appropriate test!

[ericbottard]

Merged to main as 8e8654e