Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support function calling for reflection Method with multiple arguments #1099

Closed

Conversation

tzolov
Copy link
Contributor

@tzolov tzolov commented Jul 23, 2024

feat: Add function calling support to invoke methods with dynamic arguments and return values

This change enables more flexible integration between Spring AI and LLM function
calling capabilities while maintaining type safety and ease of use.

- Add new MethodFunctionCallback class to support method invocation via reflection
 - Supports both static and non-static method calls
 - Handles multiple parameter types including primitives, objects, collections
 - Supports empty parameters and empty response
 - Auto-generates JSON schema from method parameters
 - Special handling for ToolContext parameters
 - Builder pattern for easy configuration

- Add comprehensive unit tests for MethodFunctionCallback
- Add integration tests for MethodFunctionCallback with both Anthropic and OpenAI clients
- Add jackson-module-jsonSchema dependency
- Modify FunctionCallback to check for empty tool context

Testing coverage includes:
- Static method invocation scenarios
- Non-static method calls with various parameter types
- Void return type methods
- Complex parameter types (enums, records, lists)
- Tool context handling
- Error cases and validation

KAMO030 added a commit to KAMO030/spring-ai that referenced this pull request Jul 24, 2024
-- add FunctionCallbackMethodProcessor to support parsing.
-- add the FunctionCalling annotation to declare
-- add FunctionCallbackMethodProcessorIT as test

Dependent on spring-projects#1099
KAMO030 added a commit to KAMO030/spring-ai that referenced this pull request Jul 24, 2024
-- add FunctionCallbackMethodProcessor to support parsing.
-- add the FunctionCalling annotation to declare
-- add FunctionCallbackMethodProcessorIT as test

Dependent on spring-projects#1099
String className = entry.getKey();
Class<?> clazz = entry.getValue();

JsonSchema schema = schemaGen.generateSchema(clazz);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking into BeanOutputConverter we could use the following code to reuse the "victools" json schema that I believe is what supports the 2020_12 draft as standard jackson doesn't support that version of the schema. If jackson now supports it, that would be great.

		JacksonModule jacksonModule = new JacksonModule();
		SchemaGeneratorConfigBuilder configBuilder = new SchemaGeneratorConfigBuilder(DRAFT_2020_12, PLAIN_JSON)
			.with(jacksonModule);
		SchemaGeneratorConfig config = configBuilder.build();
		SchemaGenerator generator = new SchemaGenerator(config);
		JsonNode jsonNode = generator.generateSchema(this.typeRef.getType());
        ...  convert node to string..


@markpollack markpollack self-assigned this Jul 24, 2024
@csterwa csterwa mentioned this pull request Sep 12, 2024
10 tasks
@tzolov tzolov force-pushed the function-calling-method-utils branch from f47fe19 to 8d11a01 Compare October 4, 2024 13:10
@markpollack markpollack added this to the 1.0.0-M4 milestone Oct 23, 2024
@tzolov tzolov force-pushed the function-calling-method-utils branch from 8d11a01 to 73205e4 Compare November 8, 2024 18:10
…uments and return values

This change enables more flexible integration between Spring AI and LLM function
calling capabilities while maintaining type safety and ease of use.

- Add new MethodFunctionCallback class to support method invocation via reflection
 - Supports both static and non-static method calls
 - Handles multiple parameter types including primitives, objects, collections
 - Supports empty parameters and empty response
 - Auto-generates JSON schema from method parameters
 - Special handling for ToolContext parameters
 - Builder pattern for easy configuration

- Add comprehensive unit tests for MethodFunctionCallback
- Add integration tests for MethodFunctionCallback with both Anthropic and OpenAI clients
- Add jackson-module-jsonSchema dependency
- Modify FunctionCallback to check for empty tool context

Testing coverage includes:
- Static method invocation scenarios
- Non-static method calls with various parameter types
- Void return type methods
- Complex parameter types (enums, records, lists)
- Tool context handling
- Error cases and validation
@tzolov tzolov force-pushed the function-calling-method-utils branch from 73205e4 to f302dc0 Compare November 9, 2024 19:17
@markpollack
Copy link
Member

This looks amazing. I'm havingsome odd issues that seem to be related to classpath.

when building and then running ./mvnw verify -Pintegration-tests -pl models/spring-ai-openai I get a test failure

[INFO] Running org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT
[ERROR] Tests run: 6, Failures: 1, Errors: 5, Skipped: 0, Time elapsed: 0.029 s <<< FAILURE! -- in org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT
[ERROR] org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodTurnLightNoResponse -- Time elapsed: 0.005 s <<< ERROR!
java.lang.NoClassDefFoundError: org/springframework/ai/model/function/MethodFunctionCallback
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodTurnLightNoResponse(OpenAiChatClientMethodFunctionCallbackIT.java:145)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.ClassNotFoundException: org.springframework.ai.model.function.MethodFunctionCallback
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	... 4 more

[ERROR] org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherToolContext -- Time elapsed: 0.002 s <<< ERROR!
java.lang.NoClassDefFoundError: org/springframework/ai/model/function/MethodFunctionCallback
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherToolContext(OpenAiChatClientMethodFunctionCallbackIT.java:196)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.ClassNotFoundException: org.springframework.ai.model.function.MethodFunctionCallback
	... 4 more

[ERROR] org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodNoParameters -- Time elapsed: 0.002 s <<< ERROR!
java.lang.NoClassDefFoundError: org/springframework/ai/model/function/MethodFunctionCallback
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodNoParameters(OpenAiChatClientMethodFunctionCallbackIT.java:246)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.ClassNotFoundException: org.springframework.ai.model.function.MethodFunctionCallback
	... 4 more

[ERROR] org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherNonStatic -- Time elapsed: 0.002 s <<< ERROR!
java.lang.NoClassDefFoundError: org/springframework/ai/model/function/MethodFunctionCallback
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherNonStatic(OpenAiChatClientMethodFunctionCallbackIT.java:171)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.ClassNotFoundException: org.springframework.ai.model.function.MethodFunctionCallback
	... 4 more

[ERROR] org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherToolContextButNonContextMethod -- Time elapsed: 0.005 s <<< FAILURE!
java.lang.AssertionError: Configured method does not accept ToolContext as input parameter!: unexpected exception type thrown; expected:<java.lang.IllegalArgumentException> but was:<java.lang.NoClassDefFoundError>
	at org.junit.Assert.assertThrows(Assert.java:1020)
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherToolContextButNonContextMethod(OpenAiChatClientMethodFunctionCallbackIT.java:221)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.NoClassDefFoundError: org/springframework/ai/model/function/MethodFunctionCallback
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.lambda$methodGetWeatherToolContextButNonContextMethod$0(OpenAiChatClientMethodFunctionCallbackIT.java:224)
	at org.junit.Assert.assertThrows(Assert.java:1001)
	... 4 more
Caused by: java.lang.ClassNotFoundException: org.springframework.ai.model.function.MethodFunctionCallback
	... 6 more

[ERROR] org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherStatic -- Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: org/springframework/ai/model/function/MethodFunctionCallback
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodGetWeatherStatic(OpenAiChatClientMethodFunctionCallbackIT.java:122)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.ClassNotFoundException: org.springframework.ai.model.function.MethodFunctionCallback
	... 4 more


and when running it in intellij it gives


java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/jsonSchema/JsonSchemaGenerator

	at org.springframework.ai.model.function.MethodFunctionCallback.generateJsonSchema(MethodFunctionCallback.java:184)
	at org.springframework.ai.model.function.MethodFunctionCallback.<init>(MethodFunctionCallback.java:108)
	at org.springframework.ai.model.function.MethodFunctionCallback$Builder.build(MethodFunctionCallback.java:305)
	at org.springframework.ai.openai.chat.client.OpenAiChatClientMethodFunctionCallbackIT.methodTurnLightNoResponse(OpenAiChatClientMethodFunctionCallbackIT.java:149)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.module.jsonSchema.JsonSchemaGenerator
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	... 7 more

when running OpenAiChatClientMethodFunctionCallbackIT.methodTurnLightNoResponse

@tzolov
Copy link
Contributor Author

tzolov commented Nov 10, 2024

@markpollack thanks for reviewing it.
I guess the errors you observer are due to the need to rebuild the entire project first: e.g. ./mvnw clean install -DskipTests or ./mvnw clean install -DskipTests -U.
Then the ./mvnw verify -Pintegration-tests -pl models/spring-ai-openai should work. I guess something similar is happening with the IDE.

@markpollack
Copy link
Member

The -U seems to have helped with the maven build, so that is passing now. For IntelliJ I had to do the 'repair ide' steps and eventually it worked. Odd.

}
catch (Exception e) {
ReflectionUtils.handleReflectionException(e);
return null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why return null?

@markpollack
Copy link
Member

merged in 5f6b892

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants