Node 18 UND_ERR_HEADERS_TIMEOUT error encountered when calling Ollama invoke, even when overriding keepAlive default

[elliotrodriguez]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const ollama = new Ollama({
    baseUrl: "http://127.0.0.1:11434",
    model: "phi3",
    options: {
        "temperature": 1
    }
  });
  
  ollama.invoke(prompt).then(res => ...

Description

I am running a local Ollama server with hardware resource constraints, but I have verified the application has installed correctly and works from the CLI on this hardware, and the issue I have encountered does not occur on faster hardware.

An associated nodejs application I wrote that simply invokes the model using langchain fails with an UND_ERR_HEADERS_TIMEOUT error complaining the fetch failed. And the stack points to line 12 in dist/utils/ollama.cjs, which is where the fetch occurs.

My understanding based on related issues I've found during research is that this timeout can be extended using OLLAMA_KEEP_ALIVE, specifically this here:
https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-completion

Assuming my understanding is correct, langchain does not honor a value I use to override this, neither inline with the LLM call nor when I set the OLLAMA_KEEP_ALIVE environment variable. The error surfaces after 5 minutes have passed from when I call invoke, which appears to be the default set here and also in the associated embeddings file.

So assuming that the keep_alive property is indeed what I need to ensure the model has sufficient time under hardware constraints to complete a response and get a completion, what am I doing wrong? Thanks.

System Info

node 18.20.3
Debian Linux
Langchain v 0.2.9

[dosubot[bot]]

Hello, @elliotrodriguez! I'm a bot that can help you with bug fixes, questions, and becoming a contributor. I'm here to assist you while waiting for a human maintainer.

To properly override the keepAlive timeout value in LangChain to prevent the UND_ERR_HEADERS_TIMEOUT error when calling Ollama.invoke, you need to set the keepAlive parameter when creating an instance of the Ollama class. Here is an example of how to do this:

const ollama = new Ollama({
  baseUrl: "http://127.0.0.1:11434",
  model: "phi3",
  keepAlive: "10m", // Set the keepAlive timeout to 10 minutes
  options: {
    temperature: 1
  }
});

ollama.invoke(prompt).then(res => {
  // Handle the response
}).catch(err => {
  // Handle the error
});

In this example, the keepAlive parameter is set to 10m, which overrides the default value of 5m. Adjust the value as needed based on your hardware constraints ^[1][2].

Additionally, ensure that you are not setting the OLLAMA_KEEP_ALIVE environment variable, as it might not be honored by LangChain. Instead, set the keepAlive parameter directly in the Ollama class instance as shown above.

To continue talking to Dosu, mention @dosu.

[elliotrodriguez]

This does not work. I've tried this before as well and neglected to mention it in my question:

keepAlive:"30m"

again based on my understanding of what this parameter does in context of the fetch, the above value should provide 30 minutes before timeout, but it reliably times out after 5 minutes. I've personally timed and observed the response.

TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at async createOllamaStream (/home/elliot/recipes/node_modules/@langchain/community/dist/utils/ollama.cjs:12:22)
    at async createOllamaGenerateStream (/home/elliot/recipes/node_modules/@langchain/community/dist/utils/ollama.cjs:57:5)
    at async Ollama._streamResponseChunks (/home/elliot/recipes/node_modules/@langchain/community/dist/llms/ollama.cjs:346:26)
    at async Ollama._call (/home/elliot/recipes/node_modules/@langchain/community/dist/llms/ollama.cjs:376:26)
    at async Promise.all (index 0)
    at async Ollama._generate (/home/elliot/recipes/node_modules/@langchain/core/dist/language_models/llms.cjs:359:29)
    at async Ollama._generateUncached (/home/elliot/recipes/node_modules/@langchain/core/dist/language_models/llms.cjs:173:26)
    at async Ollama.invoke (/home/elliot/recipes/node_modules/@langchain/core/dist/language_models/llms.cjs:35:24) {
  cause: HeadersTimeoutError: Headers Timeout Error
      at Timeout.onParserTimeout [as callback] (node:internal/deps/undici/undici:9117:32)
      at Timeout.onTimeout [as _onTimeout] (node:internal/deps/undici/undici:7148:17)
      at listOnTimeout (node:internal/timers:569:17)
      at process.processTimers (node:internal/timers:512:7) {
    code: 'UND_ERR_HEADERS_TIMEOUT'
  }
}

There is a related issue here that mentions a fix:
nodejs/undici#1531

Which was not effective until I updated to node 20.

So it would seem the solution is, for those who can, upgrade to node 20 until/unless something is done to address the fetch bug in node 18.