An exception occurred during handling of failed request

[HJK181]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/http (HttpCrawler)

Issue description

My HttpCrawler fails with

DEBUG HttpCrawler: Crawled 18/547 pages, 0 failed requests, desired concurrency 2.
ERROR HttpCrawler: An exception occurred during handling of failed request. This places the crawler and its underlying storages into an unknown state and crawling will be terminated. 
  Handling request failure of https://cc-stage-shopware.kleerly.de/epson/stylus/ (3FHkwFLmzy6kAAL) timed out after 300 seconds.
      at Timeout._onTimeout (/home/myuser/node_modules/@apify/timeout/cjs/index.cjs:64:68)
      at listOnTimeout (node:internal/timers:581:17)
      at process.processTimers (node:internal/timers:519:7)
ERROR HttpCrawler:AutoscaledPool: runTaskFunction failed.
  Handling request failure of https://cc-stage-shopware.kleerly.de/epson/stylus/ (3FHkwFLmzy6kAAL) timed out after 300 seconds.
      at Timeout._onTimeout (/home/myuser/node_modules/@apify/timeout/cjs/index.cjs:64:68)
      at listOnTimeout (node:internal/timers:581:17)
      at process.processTimers (node:internal/timers:519:7)

The only kind of errors logged before this happened were (19 times according to SDK_CRAWLER_STATISTICS_6.json):

ERROR Request failed with Detected a session error, rotating session... 
Proxy responded with 500 Unable to connect: 0 bytes.

Below is the first 100 bytes of the proxy response body:

Code sample

const crawler = new HttpCrawler(
      {
        maxConcurrency: 2,
        maxRequestsPerMinute: 180,
        ...options,
        proxyConfiguration,
        useSessionPool: true,
        persistCookiesPerSession: true,
        additionalMimeTypes: ["text/plain", "application/pdf"],
        async requestHandler({ pushData, request, response }) {
          await pushData({
            url: request.url,
            statusCode: response.statusCode,
          });
        },
        async failedRequestHandler({ pushData, request, response }) {
          log.error(`Request for URL "${request.url}" failed.`);
          await pushData({
            url: request.url,
            statusCode: response?.statusCode ?? 0,
          });
        },
        async errorHandler({ request }, { message }) {
          log.error(`Request failed with ${message}`);
          if (!request.noRetry) {
            const baseWaitTime = Math.pow(2, request.retryCount) * 1000;
            const jitter = baseWaitTime * (Math.random() - 0.5);
            const waitTime = baseWaitTime + jitter;
            await new Promise((resolve) => setTimeout(resolve, waitTime));
          }
        },
      },
      config,
    );
crawler.run(urls)

Package version

crawlee@3.11.5

Node.js version

v20.18.0

Operating system

Docker image From apify/actor-node-playwright-chrome

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response