Multi-threading

[KeenanFernandes2000]

Is there a way to configure the code to cater to multiple user at the same time, I have server that can support the load however when multiple people try to access the endpoint it goes in a sequential mode and not multi-threaded.

[jakobhoeg]

Hey, @KeenanFernandes2000 !

Yes, it is possible in v.0.1.33 of Ollama. You need to set some environmental variables. Check out this link, specifically under Experimental concurrency features.

OLLAMA_NUM_PARALLEL: Handle multiple requests simultaneously for a single model
OLLAMA_MAX_LOADED_MODELS: Load multiple models simultaneously

Example:

OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4

[KeenanFernandes2000]

@jakobhoeg thank you for the quick response and with the source. Appreciate it 🙏