getting a polling error: aggregate error

[mildly2late]

Hi im getting this error when deploy my telegram bot, has anyone seen this before? ive used this package for multiple computers and its the first time im encountering this

Unhandled rejection RequestError: AggregateError
at new RequestError (C:\Users\USER\node_modules\request-promise-core\lib\errors.js:14:15)
at plumbing.callback (C:\Users\USER\node_modules\request-promise-core\lib\plumbing.js:87:29)
at Request.RP$callback [as _callback] (C:\Users\USER\node_modules\request-promise-core\lib\plumbing.js:46:31)
at self.callback (C:\Users\USER\node_modules\request\request.js:185:22)
at Request.emit (node:events:514:28)
at Request.onRequestError (C:\Users\USER\node_modules\request\request.js:877:8)
at ClientRequest.emit (node:events:514:28)
at TLSSocket.socketErrorListener (node:_http_client:495:9)
at TLSSocket.emit (node:events:514:28)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

[windkh]

I guess I remember having seen a similar thing when connection was lost or rejected due to network failure. Not 100% sure

[mildly2late]

thanks for the reply, i tried running on a test PC in the same network before-hand and seems to be fine. just hoping that i dont have to swap out the whole thing

[E1cid]

Seeing similar issue since update to 15.1.9

Unhandled rejection Error: ETELEGRAM: 409 Conflict: terminated by other getUpdates request; make sure that only one bot instance is running
    at /data/data/com.termux/files/home/.node-red/node_modules/node-telegram-bot-api/src/telegram.js:299:15
    at tryCatcher (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/async.js:102:5)
    at Async.drainQueues [as _onImmediate] (/data/data/com.termux/files/home/.node-red/node_modules/bluebird/js/release/async.js:15:14)
    at process.processImmediate (node:internal/timers:478:21)

If i turn poling off in config the error stops

[windkh]

This indicates that you used the same token in several places. The server does not like if the messages are polled by two parties

[windkh]

please try with new version 16.0.0

[duycop]

node-red-contrib-telegrambot
version: 16.0.1

• node "receiver" polling -> error -> polling ->error : loop forever
• node "sender" => getting a error: **EFATAL: AggregateError**

[windkh]

@duycop can you reproduce it?

[ivanjh]

tl;dr Add to environment variable NODE_OPTIONS switch --no-network-family-autoselection to Node Red and restart

For me it was this Nodejs bug when connection via ipv4 is slow (>250ms) & ipv6 is unrouted. (nodejs/node#54359)

How did I end up here?

I had this immediately after a node-red upgrade (v2 to v3 to v4) and node-red-contrib-telegrambot upgrade (v14.? to v16.1.3). This would have upgraded the Node version and introduced the bug.

How did I find this?

Logs were very poor. node-telegram-bot-api has a truly horrid concept of FatalError. FatalError prefixes the message from a source error, copies the source error's stack - and discards the rest. Leaving nothing of interest to show in logs.

I put this disgrace in patch-efatal.js to replace FatalError with a slightly better one:

let tgbe=require('node-telegram-bot-api/src/errors');
class FatalError extends tgbe.BaseError {
  constructor(data) {
    const error = (typeof data === 'string') ? null : data;
    const message = error ? error.message : data;
    super('SLIGHTLYBETTEREFATAL', message);
    if (error) this.stack = error.stack;
    if (error) this.cause = error;
  }
};
tgbe.FatalError=FatalError;

And logged a deeper inspect of the error.

--- 99-telegrambot.js
+++ 99-telegrambot.js
@@ -3,8 +3,6 @@
 // Avoid that node-telegram-bot-api will enable automatci promise cancellation (fix for 0.30.0 api)
 process.env['NTBA_FIX_319'] = 1;

+require('./patch-efatal')
+
 module.exports = function (RED) {
     'use strict';

@@ -442,8 +440,6 @@

                 if (self.verbose) {
                     self.warn(error.message);
+                    self.warn(require("node:util").inspect(error,{depth:5}));
                 }

                 let stopPolling = false;

At this point, I was getting logs showing AggregateError for IPv4 timeout and IPv6 unreachable.

AggregateError [ETIMEDOUT]:
    at internalConnectMultiple (node:net:1139:18)
    at afterConnectMultiple (node:net:1712:7) {
  code: 'ETIMEDOUT',
  [errors]: [
    Error: connect ETIMEDOUT 149.154.167.220:443
        at createConnectionError (node:net:1675:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1734:38)
        at listOnTimeout (node:internal/timers:596:11)
        at process.processTimers (node:internal/timers:529:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '149.154.167.220',
      port: 443
    },
    Error: connect ENETUNREACH 2001:67c:4e8:f004::9:443
        at createConnectionError (node:net:1675:14)
        at afterConnectMultiple (node:net:1705:16) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2001:67c:4e8:f004::9',
      port: 443
    }
  ]
}

It was "timing out" ipv4 almost immediately - sub-second. Which didn't feel like a "normal" http connection timeout (I connect around the globe often enough to know some places are 1/2 a second just to tcp connect). And I could curl that ipv4 address without any connection issue. It was also failing an ipv6 connection at the same time. That was so weird that I felt it wasn't done by "application" code. A little Google search around AggregateError ETIMEDOUT ENETUNREACH found the rest.

And yes, I am over 250ms RTT to telegram api servers:

$ sudo ping api.telegram.org
PING api.telegram.org (149.154.167.220) 56(84) bytes of data.
64 bytes from 149.154.167.220 (149.154.167.220): icmp_seq=1 ttl=45 time=286 ms
64 bytes from 149.154.167.220 (149.154.167.220): icmp_seq=2 ttl=45 time=282 ms
64 bytes from 149.154.167.220 (149.154.167.220): icmp_seq=3 ttl=45 time=286 ms

Added environment variable NODE_OPTIONS with a value of --no-network-family-autoselection and it immediately worked.
In theory --network-family-autoselection-attempt-timeout=500 would also work but I'm just going to disable it and move on.

[windkh]

@ivanjh Thanks for your great work!
Is there something that I can add to the code that you found out in case somebody else wants to debug a similar problem in future?

[ivanjh]

@ivanjh Thanks for your great work! Is there something that I can add to the code that you found out in case somebody else wants to debug a similar problem in future?

A quick search in the issues on this repo shows a fair bit of EFATAL. All of which, aren't going to give a detailed error message.

Step 1:

We need to log the error in a more detailed fashion. The error.message isn't enough... all the detail is in the causing error (and .errors[] for AggregateError). So instead of self.warn(error.message); something like self.warn(require("node:util").inspect(error,{depth:5})); to dump more detail.

Step 2 (best & slowest):

The best way would be for the node-telegram-bot-api library to alter FatalError to keep the original error in .cause (https://nodejs.org/api/errors.html#errorcause). I'm not immediately aware of a reason not to. But that's relying on an external party. That repo has a last merged PR 7 months ago, and last commit is 5 months ago. It might take a while to happen. But it'd be worth raising a PR to start a discussion. Here's one I prepared earlier yagop/node-telegram-bot-api#1257

or Step 2 (nasty & quick):

You could do exactly what I did and override their FatalError Class with a patched version that sets .cause. Of course, you're fiddling with the internals of another library - and would likely want to lock to a specific version ("node-telegram-bot-api": "0.66.0", instead of "node-telegram-bot-api": "^0.66.0",) - to not get a future conflicting update. You could do that today and immediately allow your users to get better messages.

or Step 2 (both)

Do both. Immediately release a patching version. Then, raise a PR to retain the causing error on node-telegram-bot-api. Undo the patching once the PR is merged & released.

[windkh]

@ivanjh I applied you patch: thanks alot. Waiting for your pull request in node-telegrambot-api:

[ivanjh]

@windkh Nice work. This will allow anyone getting EFATAL errors to have more detailed logging to include when reporting issues.

I've been using node-red-contrib-telegrambot for years, and the first problem I've had is a nodejs bug somewhat obscured by a dependency library - that's a pretty good. You've done a great job - thank you.

[windkh]

@ivanjh thanks, would not have been possible without your help :)