ECONNRESET error from the proxy inside EKS kubernetes clusters

[josephkurian]

yarn why http-proxy-middleware OR npm ls http-proxy-middleware output (mask private folder names with *****)

This is from inside the pod that runs backstage that relies on HPM
root@backstage-7b7475f455-bv7pp:/app# yarn why http-proxy-middleware
yarn why v1.22.5
[1/4] Why do we have the module "http-proxy-middleware"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "http-proxy-middleware@0.19.1"
info Reasons this module exists
   - "_project_#@backstage#cli#webpack-dev-server" depends on it
   - Hoisted from "_project_#@backstage#cli#webpack-dev-server#http-proxy-middleware"
=> Found "@backstage/plugin-proxy-backend#http-proxy-middleware@0.19.2"
info This module exists because "_project_#backend#@backstage#plugin-proxy-backend" depends on it.
info Disk size without dependencies: "80KB"
info Disk size with unique dependencies: "5.44MB"
info Disk size with transitive dependencies: "9.89MB"
info Number of shared dependencies: 32
Done in 4.16s.

• I understand project setup issues should be asked on StackOverflow or in GitHub Discussions.
• I updated to latest http-proxy-middleware.

Describe the bug (be clear and concise)

We are developing a backstage.io backed developer portal with a custom backend plugin that invokes Rest API calls to some of our backend services. We have configured the HPM proxy to make the calls to these APIs as advised in the docs : https://backstage.io/docs/plugins/proxying ,
We have deployed the portal in an EKS cluster using the Postgres backend and an nginx ingress based AWS network loadbalancer to serve as the entry point to the portal.
I am facing an issue where about 30-40 % of the times , the backend API calls made by the proxy loads really slow and from the backstage pod logs I notice that whenever this happens, I get the error

proxy error [HPM] Error occurred while trying to proxy request from to (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors) type=plugin

One thing I noted is that almost always when I get this error, the time it takes to get the response is exactly one min.

EKS K8 version: 1.17

Inside the docker container:

root@backstage-c7786ddfb-ck7b6:/app# npm --version
6.14.14
root@backstage-c7786ddfb-ck7b6:/app# node --version
v14.17.4
root@backstage-c7786ddfb-ck7b6:/app# npm version
{
root: '1.0.0',
npm: '6.14.14',
ares: '1.17.1',
brotli: '1.0.9',
cldr: '39.0',
icu: '69.1',
llhttp: '2.1.3',
modules: '83',
napi: '8',
nghttp2: '1.42.0',
node: '14.17.4',
openssl: '1.1.1k',
tz: '2021a',
unicode: '13.0',
uv: '1.41.0',
v8: '8.4.371.23-node.76',
zlib: '1.2.11'
}

Wondering if anyone has faced similar issues and if I can get some help?

Step-by-step reproduction instructions

1. Load the page several times and the error shows up 30-40 % of the time

Expected behavior (be clear and concise)

ECONNRESET should not happen and we should not see any latency while invoking the backend API calls.

What http-proxy-middleware configuration are you using?

proxy:
  '/path':
    target: <apiendpoint>
    changeOrigin: true
    allowedHeaders: ['Authorization']
    allowedHeaders: [ 'Authorization' ]
    headers:
      Connection: 'keep-alive'

What OS/version and node/version are you seeing the problem?

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster

4.14.209-160.335.amzn2.x86_64

node --version
v14.17.4

Additional context (optional)

No response

[chimurai]

Issue seems to be similar to these reports: webpack/webpack-dev-server#1642

Could you try updating to v2.0.6? This version contains a fix for the ECONNRESET error.

[chimurai]

Feel free to reopen issue if problem persists