feat(ollama): add retry template integration to OllamaChatModel #1852
Conversation
apappascs
commented
Dec 2, 2024
apappascs
commented
- Integrated RetryTemplate for API calls in OllamaChatModel.
- Ensured consistent retry logic across all model classes.
|
Thanks for your contribution! |
| @@ -198,7 +213,8 @@ public Flux<ChatResponse> stream(Prompt prompt) { | |||
|
|
|||
| observation.parentObservation(contextView.getOrDefault(ObservationThreadLocalAccessor.KEY, null)).start(); | |||
|
|
|||
| Flux<OllamaApi.ChatResponse> ollamaResponse = this.chatApi.streamingChat(request); | |||
| Flux<OllamaApi.ChatResponse> ollamaResponse = this.retryTemplate | |||
There was a problem hiding this comment.
When in streaming mode, we don't use the RetryTemplate as it's not gonna work with the reactive streams. There is pending design on how to configure retries for streaming. You can follow this issue for updates: #1193
There was a problem hiding this comment.
Thank you @ThomasVitale for pointing this out.
There was a problem hiding this comment.
Adding some context to this issue: a
RetryTemplateis used to add retry capabilities to the blocking model calls. At first, the same strategy was used for the streaming calls. After realising that it wasn't working as expected in a reactive stream, the usage ofRetryTemplatestreaming calls has been removed. In this issue, we would need to come up with a design for a retry feature to adopt in all streaming implementations across the different model providers.
Its still in use in the code tho: https://github.com/search?q=repo%3Aspring-projects%2Fspring-ai+%22Flux%3CChatCompletionChunk%3E%22+this.retryTemplate+execute&type=code
should I open a PR removing it?
| @@ -124,6 +125,7 @@ public OllamaChatModel ollamaChat(OllamaApi ollamaApi) { | |||
| return OllamaChatModel.builder() | |||
| .withOllamaApi(ollamaApi) | |||
| .withDefaultOptions(OllamaOptions.create().withModel(MODEL).withTemperature(0.9)) | |||
| .withRetryTemplate(RetryUtils.DEFAULT_RETRY_TEMPLATE) | |||
There was a problem hiding this comment.
Is this needed? If I read correctly, RetryUtils.DEFAULT_RETRY_TEMPLATE is provided by default if no explicit RestTemplate object is provided, so there shouldn't be a need for adding this line.
Same question for all the other places where this line was added.
There was a problem hiding this comment.
in comparison to other tests on ollama the model is configured inside the test https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/test/java/org/springframework/ai/ollama/OllamaChatModelIT.java#L244
apappascs
force-pushed
the
feature/add-retry-to-ollama-model
branch
from
3a28241 to
729b807
Compare
apappascs
force-pushed
the
feature/add-retry-to-ollama-model
branch
from
729b807 to
55d38fd
Compare
