Trying to use token-window memory type with easy rag results in exception

[edeandrea]

I have an app (https://github.com/edeandrea/java-ai-playground/tree/quarkus) that if I try to set quarkus.langchain4j.chat-memory.type=token-window I end up getting a big fat stack trace (see below). The app uses the easy-rag extension.

1. Clone https://github.com/edeandrea/java-ai-playground
2. Check out the quarkus branch
3. Make sure Ollama is running
4. Run ./mvnw clean quarkus:dev -Dquarkus.profile=ollama -Dquarkus.langchain4j.chat-memory.type=token-window
5. See the stack trace:

2024-11-01 08:07:22,362 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkus.arc.deployment.ArcProcessor#validate threw an exception: jakarta.enterprise.inject.spi.DeploymentException: jakarta.enterprise.inject.UnsatisfiedResolutionException: Unsatisfied dependency for type dev.langchain4j.model.Tokenizer and qualifiers [@jakarta.enterprise.inject.Default]
        - synthetic injection point
        - declared on SYNTHETIC bean [types=[dev.langchain4j.memory.chat.ChatMemoryProvider, java.lang.Object], qualifiers=[@Default, @Any], target=n/a]
        at io.quarkus.arc.processor.BeanDeployment.processErrors(BeanDeployment.java:1576)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:338)
        at io.quarkus.arc.processor.BeanProcessor.initialize(BeanProcessor.java:178)
        at io.quarkus.arc.deployment.ArcProcessor.validate(ArcProcessor.java:492)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        at org.jboss.threads.JBossThread.run(JBossThread.java:499)
Caused by: jakarta.enterprise.inject.UnsatisfiedResolutionException: Unsatisfied dependency for type dev.langchain4j.model.Tokenizer and qualifiers [@jakarta.enterprise.inject.Default]
        - synthetic injection point
        - declared on SYNTHETIC bean [types=[dev.langchain4j.memory.chat.ChatMemoryProvider, java.lang.Object], qualifiers=[@Default, @Any], target=n/a]
        at io.quarkus.arc.processor.Beans.resolveInjectionPoint(Beans.java:546)
        at io.quarkus.arc.processor.BeanInfo.init(BeanInfo.java:698)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:323)
        ... 12 more

        at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:354)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:272)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:62)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.firstStart(IsolatedDevModeMain.java:89)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:428)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:55)
        at io.quarkus.bootstrap.app.CuratedApplication.runInCl(CuratedApplication.java:138)
        at io.quarkus.bootstrap.app.CuratedApplication.runInAugmentClassLoader(CuratedApplication.java:93)
        at io.quarkus.deployment.dev.DevModeMain.start(DevModeMain.java:131)
        at io.quarkus.deployment.dev.DevModeMain.main(DevModeMain.java:62)
Caused by: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkus.arc.deployment.ArcProcessor#validate threw an exception: jakarta.enterprise.inject.spi.DeploymentException: jakarta.enterprise.inject.UnsatisfiedResolutionException: Unsatisfied dependency for type dev.langchain4j.model.Tokenizer and qualifiers [@jakarta.enterprise.inject.Default]
        - synthetic injection point
        - declared on SYNTHETIC bean [types=[dev.langchain4j.memory.chat.ChatMemoryProvider, java.lang.Object], qualifiers=[@Default, @Any], target=n/a]
        at io.quarkus.arc.processor.BeanDeployment.processErrors(BeanDeployment.java:1576)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:338)
        at io.quarkus.arc.processor.BeanProcessor.initialize(BeanProcessor.java:178)
        at io.quarkus.arc.deployment.ArcProcessor.validate(ArcProcessor.java:492)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        at org.jboss.threads.JBossThread.run(JBossThread.java:499)
Caused by: jakarta.enterprise.inject.UnsatisfiedResolutionException: Unsatisfied dependency for type dev.langchain4j.model.Tokenizer and qualifiers [@jakarta.enterprise.inject.Default]
        - synthetic injection point
        - declared on SYNTHETIC bean [types=[dev.langchain4j.memory.chat.ChatMemoryProvider, java.lang.Object], qualifiers=[@Default, @Any], target=n/a]
        at io.quarkus.arc.processor.Beans.resolveInjectionPoint(Beans.java:546)
        at io.quarkus.arc.processor.BeanInfo.init(BeanInfo.java:698)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:323)
        ... 12 more

        at io.quarkus.builder.Execution.run(Execution.java:124)
        at io.quarkus.builder.BuildExecutionBuilder.execute(BuildExecutionBuilder.java:79)
        at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:161)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:350)
        ... 9 more
Caused by: jakarta.enterprise.inject.spi.DeploymentException: jakarta.enterprise.inject.UnsatisfiedResolutionException: Unsatisfied dependency for type dev.langchain4j.model.Tokenizer and qualifiers [@jakarta.enterprise.inject.Default]
        - synthetic injection point
        - declared on SYNTHETIC bean [types=[dev.langchain4j.memory.chat.ChatMemoryProvider, java.lang.Object], qualifiers=[@Default, @Any], target=n/a]
        at io.quarkus.arc.processor.BeanDeployment.processErrors(BeanDeployment.java:1576)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:338)
        at io.quarkus.arc.processor.BeanProcessor.initialize(BeanProcessor.java:178)
        at io.quarkus.arc.deployment.ArcProcessor.validate(ArcProcessor.java:492)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        at org.jboss.threads.JBossThread.run(JBossThread.java:499)
Caused by: jakarta.enterprise.inject.UnsatisfiedResolutionException: Unsatisfied dependency for type dev.langchain4j.model.Tokenizer and qualifiers [@jakarta.enterprise.inject.Default]
        - synthetic injection point
        - declared on SYNTHETIC bean [types=[dev.langchain4j.memory.chat.ChatMemoryProvider, java.lang.Object], qualifiers=[@Default, @Any], target=n/a]
        at io.quarkus.arc.processor.Beans.resolveInjectionPoint(Beans.java:546)
        at io.quarkus.arc.processor.BeanInfo.init(BeanInfo.java:698)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:323)
        ... 12 more

[jmartisk]

The javadoc for TOKEN_WINDOW says

If {@code token-window} is used, then the application must also provide a bean of type {@link Tokenizer}.

I assume you should create a CDI producer of Tokenizer that returns the same tokenizer that the chat model is using.. But we don't seem to have any examples :/

[edeandrea]

I assume you should create a CDI producer of Tokenizer that returns the same tokenizer that the chat model is using.. But we don't seem to have any examples :/

I scoured and couldn't find anything. I don't even really know which tokenizer to use, since I'm using easy-rag and everything is being done for me :) Shouldn't the tokenizer setup be done too?

[jmartisk]

Quarkus doesn't know what tokenizer is used under the hood when you're calling a remote model over HTTP (so you don't actually tokenize messages in the app's JVM), so that is tricky

[edeandrea]

I see there is an OpenAiTokenizer and a HuggingFaceTokenizer. Which one would I want if I was using Ollama with Llama 3.2?

[edeandrea]

I tried to add this:

@Dependent
public class LangChain4jTokenizerConfig {
	@Produces
	@ApplicationScoped
	@UnlessBuildProfile("ollama")
	public Tokenizer openAITokenizer(@ConfigProperty(name = "quarkus.langchain4j.openai.chat-model.model-name") String modelName) {
		return new OpenAiTokenizer(modelName);
	}

	@Produces
	@ApplicationScoped
	@IfBuildProfile("ollama")
	public Tokenizer ollamaTokenizer() {
		return new HuggingFaceTokenizer();
	}
}

and then re-run with

quarkus.langchain4j.chat-memory.type=token-window
quarkus.langchain4j.chat-memory.token-window.max-tokens=1000

while hooked up to OpenAI, but I started seeing this:

Exception in AssistantService.java:19
          17  
          18      public String chat(String chatId, String userMessage) {
        → 19          return langChain4JAssistant.chat(chatId, userMessage);
          20      }
          21  }

: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:115)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.vaadin.hilla.EndpointInvoker.invokeVaadinEndpointMethod(EndpointInvoker.java:454)
        at com.vaadin.hilla.EndpointInvoker.invoke(EndpointInvoker.java:203)
        at com.vaadin.hilla.EndpointController.doServeEndpoint(EndpointController.java:251)
        at com.vaadin.hilla.EndpointController.serveEndpoint(EndpointController.java:199)
        at com.github.mcollovati.quarkus.hilla.QuarkusEndpointController.serveEndpoint(QuarkusEndpointController.java:79)
        at com.github.mcollovati.quarkus.hilla.QuarkusEndpointController$quarkusrestinvoker$serveEndpoint_19ea4aa8e36421f8414cd9ce9157408f2c9d0890.invoke(Unknown Source)
        at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
        at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:7)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.invokeHandler(AbstractResteasyReactiveContext.java:231)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
        at org.jboss.resteasy.reactive.server.handlers.RestInitialHandler.beginProcessing(RestInitialHandler.java:48)
        at io.quarkus.resteasy.reactive.server.servlet.runtime.ResteasyReactiveServlet.service(ResteasyReactiveServlet.java:31)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:63)
        at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
        at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:67)
        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:133)
        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:65)
        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:247)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:111)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:108)
        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$8$1.call(UndertowDeploymentRecorder.java:643)
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:152)
        at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
        at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$1.handleRequest(UndertowDeploymentRecorder.java:126)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:284)
        at io.undertow.server.DefaultExchangeHandler.handle(DefaultExchangeHandler.java:18)
        at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$4$2.run(UndertowDeploymentRecorder.java:443)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:627)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because "properties" is null
        at dev.langchain4j.model.openai.InternalOpenAiHelper.toOpenAiProperties(InternalOpenAiHelper.java:262)
        at dev.langchain4j.model.openai.InternalOpenAiHelper.toOpenAiJsonSchemaElement(InternalOpenAiHelper.java:275)
        at dev.langchain4j.model.openai.InternalOpenAiHelper.lambda$toOpenAiProperties$2(InternalOpenAiHelper.java:263)
        at java.base/java.util.HashMap.forEach(HashMap.java:1429)
        at dev.langchain4j.model.openai.InternalOpenAiHelper.toOpenAiProperties(InternalOpenAiHelper.java:262)
        at dev.langchain4j.model.openai.InternalOpenAiHelper.toOpenAiParameters(InternalOpenAiHelper.java:246)
        at dev.langchain4j.model.openai.InternalOpenAiHelper.toTool(InternalOpenAiHelper.java:204)
        at dev.langchain4j.model.openai.InternalOpenAiHelper.lambda$toTools$1(InternalOpenAiHelper.java:196)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at dev.langchain4j.model.openai.InternalOpenAiHelper.toTools(InternalOpenAiHelper.java:197)
        at dev.langchain4j.model.openai.OpenAiChatModel.generate(OpenAiChatModel.java:230)
        at dev.langchain4j.model.openai.OpenAiChatModel.generate(OpenAiChatModel.java:179)
        at dev.langchain4j.model.chat.ChatLanguageModel_XNMsOaekknG7BdNZ5YSUkjh1SqE_Synthetic_ClientProxy.generate(Unknown Source)
        at io.quarkiverse.langchain4j.runtime.aiservice.AiServiceMethodImplementationSupport.doImplement(AiServiceMethodImplementationSupport.java:254)
        at io.quarkiverse.langcruntime.aiservice.AiServiceMethodImplementationSupport.implement(AiServiceMethodImplementationSupport.java:122)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$1.apply(MethodImplementationSupportProducer.java:31)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$1.apply(MethodImplementationSupportProducer.java:28)
        at io.quarkiverse.langchain4j.runtime.aiservice.SpanWrapper.wrap(SpanWrapper.java:32)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$2.apply(MethodImplementationSupportProducer.java:40)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$2.apply(MethodImplementationSupportProducer.java:37)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1.implement(MethodImplementationSupportProducer.java:46)
        at org.vaadin.marcus.langchain4j.LangChain4jAssistant$$QuarkusImpl.chat(Unknown Source)
        at org.vaadin.marcus.langchain4j.LangChain4jAssistant$$QuarkusImpl_ClientProxy.chat(Unknown Source)
        at org.vaadin.marcus.client.AssistantService.chat(AssistantService.java:19)
        at org.vaadin.marcus.client.AssistantService_ClientProxy.chat(Unknown Source)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        ... 54 more

[jmartisk]

Hmm when looking at this, I see a NPE somewhere in the insides of upstream langchain4j, so it may be a bug there. I also found this PR langchain4j/langchain4j#1668 that seems to rework some stuff about the classes involved in it, so maybe it will be fixed in the next version?
Maybe @langchain4j knows more details...

I am not sure it will be fixed by langchain4j/langchain4j#1668 as I am not sure what exactly is happening there... It seems that there is a tool defined somewhere with some unsupported parameter types? @edeandrea could you please check what's inside the ToolParameters in InternalOpenAiHelper.toOpenAiParameters when this happens? It seems there is a parameter of type Object but without any internal properties

[edeandrea]

I will investigate this a little later this morning.

[edeandrea]

@langchain4j This is the contents of ToolParameters when the above NullPointerException happens:

ToolParameters { type = "object", properties = {newFlightDate={type=object}, firstName={type=string}, lastName={type=string}, newDepartureAirport={description=3-letter code for departure airport, type=string}, newArrivalAirport={description=3-letter code for arrival airport, type=string}, bookingNumber={type=string}}, required = [bookingNumber, firstName, lastName, newFlightDate, newDepartureAirport, newArrivalAirport] }

I traced this down to this method in InternalOpenAiHelper:

  private static dev.ai4j.openai4j.chat.JsonSchemaElement toOpenAiJsonSchemaElement(Map<String, ?> properties, boolean strict) {
        // TODO rewrite when JsonSchemaElement will be used for ToolSpecification.properties
        Object type = properties.get("type");
        String description = (String) properties.get("description");
        if ("object".equals(type)) {
            List<String> required = (List<String>) properties.get("required");
            dev.ai4j.openai4j.chat.JsonObjectSchema.Builder builder = dev.ai4j.openai4j.chat.JsonObjectSchema.builder()
                    .description(description)
this is where it blows up --->     .properties(toOpenAiProperties((Map<String, ?>) properties.get("properties"), strict)); 
            if (required != null) {
                builder.required(required);
            }
            if (strict) {
                builder
                        // when strict, all fields must be required:
                        // https://platform.openai.com/docs/guides/structured-outputs/all-fields-must-be-required
                        .required(new ArrayList<>(((Map<String, ?>) properties.get("properties")).keySet()))
                        // when strict, additionalProperties must be false:
                        // https://platform.openai.com/docs/guides/structured-outputs/additionalproperties-false-must-always-be-set-in-objects
                        .additionalProperties(false);
            }
            return builder.build();
        } else if ("array".equals(type)) {
            return dev.ai4j.openai4j.chat.JsonArraySchema.builder()
                    .description(description)
                    .items(toOpenAiJsonSchemaElement((Map<String, ?>) properties.get("items"), strict))
                    .build();
        } else if (properties.get("enum") != null) {
            return dev.ai4j.openai4j.chat.JsonEnumSchema.builder()
                    .description(description)
                    .enumValues((List<String>) properties.get("enum"))
                    .build();
        } else if ("string".equals(type)) {
            return dev.ai4j.openai4j.chat.JsonStringSchema.builder()
                    .description(description)
                    .build();
        } else if ("integer".equals(type)) {
            return dev.ai4j.openai4j.chat.JsonIntegerSchema.builder()
                    .description(description)
                    .build();
        } else if ("number".equals(type)) {
            return dev.ai4j.openai4j.chat.JsonNumberSchema.builder()
                    .description(description)
                    .build();
        } else if ("boolean".equals(type)) {
            return dev.ai4j.openai4j.chat.JsonBooleanSchema.builder()
                    .description(description)
                    .build();
        } else {
            throw new IllegalArgumentException("Unknown type " + type);
        }
    }

When it blows up, here arethe value of all the objects:

That line calls back to the toOpenAiProperties method, where properties == null.

The problem seems to be in newFlightDate={type=object}, there are no properties inside. What type do you use for it?

[edeandrea]

In this particular case I only asked the AI "I'd like to change my booking", so I haven't yet provided the assistant with the details of what i'd like to change to.

[edeandrea]

Also, the type ofnewFlightDate is java.time.LocalDate. This is the method signature of the Tool:

@Tool("""
            Modifies an existing booking.
            This includes making changes to the flight date, and the departure and arrival airports.
            """)
    public void changeBooking(
        String bookingNumber,
        String firstName,
        String lastName,
        LocalDate newFlightDate,
        @P("3-letter code for departure airport") String newDepartureAirport,
        @P("3-letter code for arrival airport") String newArrivalAirport
    )

I suspect that the code that generates ToolSpecification does not handle LocalDate properly. AFAIK, this code is in Quarkus-LC4j project, seems like ToolProcessor?

[edeandrea]

@geoand / @jmartisk it seems that io.quarkiverse.langchain4j.deployment.ToolProcessor doesn't handle the LocalDate parameter properly?

When using straight LangChain4j upstream everything seems to work fine.

quarkus-langchain4j/core/deployment/src/main/java/io/quarkiverse/langchain4j/deployment/ToolProcessor.java

Lines 418 to 510 in 4c72522

	private Iterable<JsonSchemaProperty> toJsonSchemaProperties(Type type, IndexView index, JsonSchemaProperty description) {
	DotName typeName = type.name();
	if (type.kind() == Type.Kind.WILDCARD_TYPE) {
	Type boundType = type.asWildcardType().extendsBound();
	if (boundType == null) {
	boundType = type.asWildcardType().superBound();
	}
	if (boundType != null) {
	return toJsonSchemaProperties(boundType, index, description);
	} else {
	throw new IllegalArgumentException("Unsupported wildcard type with no bounds: " + type);
	}
	}
	if (DotNames.STRING.equals(typeName) || DotNames.CHARACTER.equals(typeName)
	|| DotNames.PRIMITIVE_CHAR.equals(typeName)) {
	return removeNulls(STRING, description);
	}
	if (DotNames.BOOLEAN.equals(typeName) || DotNames.PRIMITIVE_BOOLEAN.equals(typeName)) {
	return removeNulls(BOOLEAN, description);
	}
	if (DotNames.BYTE.equals(typeName) || DotNames.PRIMITIVE_BYTE.equals(typeName)
	|| DotNames.SHORT.equals(typeName) || DotNames.PRIMITIVE_SHORT.equals(typeName)
	|| DotNames.INTEGER.equals(typeName) || DotNames.PRIMITIVE_INT.equals(typeName)
	|| DotNames.LONG.equals(typeName) || DotNames.PRIMITIVE_LONG.equals(typeName)
	|| DotNames.BIG_INTEGER.equals(typeName)) {
	return removeNulls(INTEGER, description);
	}
	// TODO put constraints on min and max?
	if (DotNames.FLOAT.equals(typeName) || DotNames.PRIMITIVE_FLOAT.equals(typeName)
	|| DotNames.DOUBLE.equals(typeName) || DotNames.PRIMITIVE_DOUBLE.equals(typeName)
	|| DotNames.BIG_DECIMAL.equals(typeName)) {
	return removeNulls(NUMBER, description);
	}
	// TODO something else?
	if (type.kind() == Type.Kind.ARRAY || DotNames.LIST.equals(typeName) || DotNames.SET.equals(typeName)) {
	ParameterizedType parameterizedType = type.kind() == Type.Kind.PARAMETERIZED_TYPE ? type.asParameterizedType()
	: null;
	Type elementType = parameterizedType != null ? parameterizedType.arguments().get(0)
	: type.asArrayType().component();
	Iterable<JsonSchemaProperty> elementProperties = toJsonSchemaProperties(elementType, index, null);
	JsonSchemaProperty itemsSchema;
	if (isComplexType(elementType)) {
	Map<String, Object> fieldDescription = new HashMap<>();
	for (JsonSchemaProperty fieldProperty : elementProperties) {
	fieldDescription.put(fieldProperty.key(), fieldProperty.value());
	}
	itemsSchema = JsonSchemaProperty.from("items", fieldDescription);
	} else {
	itemsSchema = JsonSchemaProperty.items(elementProperties.iterator().next());
	}
	return removeNulls(ARRAY, itemsSchema, description);
	}
	if (isEnum(type, index)) {
	return removeNulls(STRING, enums(enumConstants(type)), description);
	}
	if (type.kind() == Type.Kind.CLASS) {
	Map<String, Object> properties = new HashMap<>();
	ClassInfo classInfo = index.getClassByName(type.name());
	List<String> required = new ArrayList<>();
	if (classInfo != null) {
	for (FieldInfo field : classInfo.fields()) {
	String fieldName = field.name();
	Iterable<JsonSchemaProperty> fieldSchema = toJsonSchemaProperties(field.type(), index, null);
	Map<String, Object> fieldDescription = new HashMap<>();
	for (JsonSchemaProperty fieldProperty : fieldSchema) {
	fieldDescription.put(fieldProperty.key(), fieldProperty.value());
	}
	properties.put(fieldName, fieldDescription);
	}
	}
	JsonSchemaProperty objectSchema = JsonSchemaProperty.from("properties", properties);
	return removeNulls(OBJECT, objectSchema, JsonSchemaProperty.from("required", required), description);
	}
	throw new IllegalArgumentException("Unsupported type: " + type);
	}