fetch fails with internal error after 5 minutes

[oleksii-korpan]

Version

18.13.0, 19.5.0

Platform

Microsoft Windows NT 10.0.19044.0 x64

Subsystem

docker image from node:19-alpine

What steps will reproduce the bug?

Error occurres with simple get request (didn't tested it with other options), which lasts longer then 5 minutes:

let response = await fetch('http://127.0.0.1/daily-collection');

How often does it reproduce? Is there a required condition?

This error occures every time when there is no response longer then 5 minutes

What is the expected behavior?

fetch should wait as long as needed, even if response will be returned in 10+ minutes

What do you see instead?

Next error occurs when I do http request, which lasts longer than 5 minutes:

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:14152:11)
    at async Object.collect (/usr/src/app/webserver/controllers/daily-collection.js:44:24)
    at async Object.execute (/usr/src/app/webserver/controllers/daily-collection.js:13:26)
    at async Server.<anonymous> (/usr/src/app/webserver/server.js:24:9) {
  cause: HeadersTimeoutError: Headers Timeout Error
      at Timeout.onParserTimeout [as _onTimeout] (node:internal/deps/undici/undici:9185:32)
      at listOnTimeout (node:internal/timers:570:11)
      at process.processTimers (node:internal/timers:511:7) {
    code: 'UND_ERR_HEADERS_TIMEOUT'
  }
}

Additional information

No response

[bnoordhuis]

It says so why in the error message: UND_ERR_HEADERS_TIMEOUT a.k.a. timed out while waiting for response headers.

Working as intended, not a bug. I'll close this but I can convert it to a conversation if you have follow-up questions.

[oleksii-korpan]

@bnoordhuis , thank you for the response
For me this behaviour was unexpected, becaues in browser (chrome) there is no timeout (or at least its very big, because my requests succesfully wait for response more than 10 minutes).
Ok, lets say in nodejs fetch method has default timeout (5min). Is it possible to change (increase) it? I will be grateful for any hints/examples.

[bnoordhuis]

I think for fetch() the answer is "no" but node's own http.get() has configurable timeouts.

[mattfysh]

This should absolutely be documented, given the current expectation of node's native fetch implementation being a swap-in replacement for the browser implementation. I just ran into this issue while building an AWS Lambda extension layer, which requires a long-running network request that only responds when the function is next invoked. This may sometimes require the request to remain open for longer than 5 minutes between invocations.

I understand these timeouts are configured by the underlying undici implementation - which itself is currently causing a concerning level of turbulence in the global fetch API, for example: #46706

It's unfortunate that configureable timeouts were not considered when shipping global.fetch

cc @mcollina

[bnoordhuis]

It's unfortunate that configureable timeouts were not considered when shipping global.fetch

fetch() is a standard. The standard doesn't say anything about timeouts. It's up to implementers to pick reasonable defaults.

If you want more control, use node's own http client.

[mcollina]

Configuring a custom agent in undici will allow to change these defaults.

[mattfysh]

@mcollina can this be done natively? I reviewed this test and the Agent appears to be incompatible with node's native agents: https://github.com/nodejs/undici/blob/7827031d2a7fd99f8ab9d5eddce127c6f700dd35/test/fetch/fetch-timeouts.js

[AasmundN]

I am having a similar issue. I have a node server running with nodemon. If the laptop, running windows, is closed while the nodemon server is running. After waiting a while and the opening the latptop again, then this issue appears. Restarting the laptop fixes the problem.

[tonskton]

To anyone who still encounters the issue even though they don't use any VPN's or Firewall, try to change your DNS to use Manual. I change mine to :
Preferred DNS server: 8.8.8.8
Alternate DNS server: 8.8.4.4
and I can use fetch or undici library with no timeout or fetch failed issues.

[evgenyfedorchenko]

http

Thanks, you saved my life. i have used http.request, and it also worked correctly with 1h timeout

const makeHttpRequest = (options, postData) =>
new Promise((resolve, reject) => {
const req = http.request(options, (res) => {
let data = "";

  res.on("data", (chunk) => {
    data += chunk;
  });

  res.on("end", () => {
    try {
      resolve(JSON.parse(data));
    } catch (error) {
      reject(error);
    }
  });
});

req.on("error", (error) => {
  reject(error);
});

req.write(postData);
req.end();

});

[clayreimann]

@mcollina @bnoordhuis the way this seems to be handled according to MDN is with timeout/abort signals is there a reason that node chose to diverge from that standard?

Undici clearly seems capable of waiting forever if configured correctly? Is this same configuration possible without a direct dependency on undici?

In my particular case looker's sdk handles the configuration for node's fetch so I don't have a direct way to configure this even if that's possible.

[christian-bromann]

I want to kindly request to re-open this issue as I don't believe it is resolved.

Working as intended, not a bug. I'll close this but I can convert it to a conversation if you have follow-up questions.

According to previous comments, there is a hard coded limitation of 5min for Node.js native fetch function. Apparently this can be changed in the underlying implementation. There is furthermore no clear documentation that Node.js's native fetch deviates from other JavaScript runtimes when it comes to this timeout.

I would like to highlight my issue with this: we are using native fetch in WebdriverIO where we make requests to initiate automate mobile sessions using Appium. It sometimes takes longer than 5min to setup the phone correctly and allow automation on it. We have received an issue for this here: webdriverio/webdriverio#13783

The amount of linked issues clearly shows that folks have issues with this situation. Is there any contribution I can make to resolve this? I am happy to jump into the code if the team believes that just increasing this timeout would be a good approach.

[michieal]

"Working As Intended" is an annoying way of saying that you're too lazy to fix something that should be configurable, instead of hard coded. And, that's just annoying.
In case you're wondering, I am here because a web application that I run locally uses the "fetch" code... and it times out before the API call can complete.
I (too) am in favor of reopening this (maybe as a feature request?) and having the change implemented.
I'll skip all of the questions and facepalming that I have over why this is hard coded... And I will simply offer the advice of: "If you think that they won't mind -- change it, because they will mind. If you think that it's longer than you'll ever need, remember that Bill Gates once said that 640k of memory is more than anyone will ever need... and look where that got him."
So, there's my words of wisdom for the day... Please, can this be reopened and properly classified so that it becomes part of someone's to-do list?

Thank you!