Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS-Lambda] - NetworkingError: Client network socket disconnected before secure TLS connection was established #3591

Closed
robsonkades opened this issue Jan 7, 2021 · 78 comments
Assignees
Labels
bug This issue is a bug. p2 This is a standard priority issue service-api This issue is due to a problem in a service API, not the SDK implementation.

Comments

@robsonkades
Copy link

We are having problems with several of our services to access DynamoDb. This problem is intermittent, but occurs randomly several times a day at each of the services on (AWS LAMBDA).

Any suggestion?

{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "NetworkingError: Client network socket disconnected before secure TLS connection was established",
    "reason": {
        "errorType": "NetworkingError",
        "errorMessage": "Client network socket disconnected before secure TLS connection was established",
        "code": "NetworkingError",
        "message": "Client network socket disconnected before secure TLS connection was established",
        "path": null,
        "host": "dynamodb.us-east-1.amazonaws.com",
        "port": 443,
        "region": "us-east-1",
        "hostname": "dynamodb.us-east-1.amazonaws.com",
        "retryable": true,
        "time": "2021-01-06T14:25:06.323Z",
        "stack": [
            "Error: Client network socket disconnected before secure TLS connection was established",
            "    at connResetException (internal/errors.js:609:14)",
            "    at TLSSocket.onConnectEnd (_tls_wrap.js:1547:19)",
            "    at Object.onceWrapper (events.js:420:28)",
            "    at TLSSocket.emit (events.js:326:22)",
            "    at TLSSocket.EventEmitter.emit (domain.js:483:12)",
            "    at endReadableNT (_stream_readable.js:1223:12)",
            "    at processTicksAndRejections (internal/process/task_queues.js:84:21)"
        ]
    },
    "promise": {},
    "stack": [
        "Runtime.UnhandledPromiseRejection: NetworkingError: Client network socket disconnected before secure TLS connection was established",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:326:22)",
        "    at process.emit (/var/task/node_modules/source-map-support/source-map-support.js:495:21)",
        "    at processPromiseRejections (internal/process/promises.js:209:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:98:32)"
    ]
}
@robsonkades robsonkades added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 7, 2021
@gyula-s
Copy link

gyula-s commented Jan 7, 2021

I got something very similar: intermittent, similar error, different service (S3):

Error: Client network socket disconnected before secure TLS connection was established
    at connResetException (internal/errors.js:609:14)
    at TLSSocket.onConnectEnd (_tls_wrap.js:1547:19)
    at Object.onceWrapper (events.js:420:28)
    at TLSSocket.emit (events.js:326:22)
    at TLSSocket.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1223:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ECONNRESET',
  path: null,
  host: 's3.eu-west-1.amazonaws.com',
  port: 443,
  localAddress: undefined
}

@robsonkades
Copy link
Author

robsonkades commented Jan 7, 2021

It seems to be a problem at Lambda.

Our support from aws:
I also want to confirm that we are already aware of this issue in Nodejs Lambda function and can confirm that it is affecting a very small set of invocations for Nodejs runtime. There are some mitigation steps that we suggest for it, meanwhile the Lambda internal team is doing the investigation for the root cause.

@ajredniwja
Copy link
Contributor

@robsonkades, thanks for providing the update, can you share the ticket number that you have with AWS and I can look into it, would like to close the issue if the issue is not related to the SDK.

@ajredniwja ajredniwja added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Jan 19, 2021
@robsonkades
Copy link
Author

@ajredniwja, Thank you for your help!

Case 7831443621

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Jan 21, 2021
@ajredniwja
Copy link
Contributor

@robsonkades, thanks for providing that, I can reach out to the team related to that and can update once I hear back from them.

@rcoundon
Copy link

rcoundon commented Feb 2, 2021

Hi - we're also seeing some instances of this in a NodeJs lambda, are there any updates?

@robsonkades
Copy link
Author

@rcoundon No solution so far

@rcoundon
Copy link

rcoundon commented Feb 2, 2021

Ok, thanks for the update. Any ETA?

@ajredniwja
Copy link
Contributor

I was not able to get any updates as of yet, I have contacted the engineers associated and will update you guys once I have some more insight from Lambda team.

@yeukfei02
Copy link

2021-02-05T01:27:25.910Z	1f0f65f6-4194-4948-b322-981bb95c4466	ERROR	Unhandled Promise Rejection 	{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "NetworkingError: Client network socket disconnected before secure TLS connection was established",
    "reason": {
        "errorType": "NetworkingError",
        "errorMessage": "Client network socket disconnected before secure TLS connection was established",
        "code": "NetworkingError",
        "message": "Client network socket disconnected before secure TLS connection was established",
        "path": null,
        "host": "dynamodb.ap-southeast-1.amazonaws.com",
        "port": 443,
        "region": "ap-southeast-1",
        "hostname": "dynamodb.ap-southeast-1.amazonaws.com",
        "retryable": true,
        "time": "2021-02-05T01:27:25.908Z",
        "stack": [
            "Error: Client network socket disconnected before secure TLS connection was established",
            "    at connResetException (internal/errors.js:609:14)",
            "    at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19)",
            "    at Object.onceWrapper (events.js:420:28)",
            "    at TLSSocket.emit (events.js:326:22)",
            "    at TLSSocket.EventEmitter.emit (domain.js:483:12)",
            "    at endReadableNT (_stream_readable.js:1241:12)",
            "    at processTicksAndRejections (internal/process/task_queues.js:84:21)"
        ]
    },
    "promise": {},
    "stack": [
        "Runtime.UnhandledPromiseRejection: NetworkingError: Client network socket disconnected before secure TLS connection was established",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:314:20)",
        "    at process.EventEmitter.emit (domain.js:483:12)",
        "    at process.emit (/var/task/node_modules/source-map-support/source-map-support.js:495:21)",
        "    at processPromiseRejections (internal/process/promises.js:209:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:98:32)"
    ]
}

I face same error in aws lambda, I am using nodejs
looks like aws-sdk lib problem??
using "aws-sdk": "^2.835.0", try upgrade version see how

@rcoundon
Copy link

rcoundon commented Feb 5, 2021

In case it's helpful, our issue is related to writes to S3.

@Sohett
Copy link

Sohett commented Feb 8, 2021

Same for us. Seems to affect quite significantly our Lambdas. Thanks @ajredniwja for the update.

@jonmifsud
Copy link

We've also got the same issue with AppSync - happening on a somewhat sensitive service.

Any idea how we can mitigate this until there is a proper fix?

@brucemackenzie
Copy link

brucemackenzie commented Feb 25, 2021

We had the same issue reading from secretsmanager but also reading from s3 today. Our lambdas are running in a VPC so that is likely the issue in our case.
https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#vpc-samples

@jonmifsud
Copy link

@brucemackenzie the lambda functions I have do not run in a VPC so I doubt this is a VPC specific issue. The primary function where this is causing us a concern has the issue does run on a cron - every minute - so not sure if that has any impact.

However, I am possibly more concerned should this could happen on webhook requests. Especially as I may not be guaranteed a retry by a third-party service should the function fail.

@robsonkades
Copy link
Author

We don't use VPC
approximately 0.7% of requests fail.

@OliverBailey
Copy link

Seeing same issue appearing here.

@illia-sh
Copy link

illia-sh commented Apr 13, 2021

+1 same issue.
@robsonkades do you still have this issue?
This has started to appear for us recently. ( lambda deployed in a VPC, but this seems do not matter )

@birsi
Copy link

birsi commented Apr 13, 2021

+1 same issue while calling a 3rd party service via https

node v14.16.0
aws-sdk 2.804.0

@paliwalgaurav
Copy link

+1 same issue
node v14.x

2021-04-14T08:07:31.506Z 8636001a-b230-4580-828b-966334e116ee INFO came to callback Error: Client network socket disconnected before secure TLS connection was established at connResetException (internal/errors.js:607:14) at TLSSocket.onConnectEnd (_tls_wrap.js:1544:19) at TLSSocket.emit (events.js:327:22) at endReadableNT (internal/streams/readable.js:1327:12) at processTicksAndRejections (internal/process/task_queues.js:80:21) { code: 'ECONNRESET', path: null, host: 'cognito-idp.us-east-2.amazonaws.com', port: 443, localAddress: undefined } undefined

@robsonkades
Copy link
Author

@illia-sh yes,
still with the same problem

@technicallyfeasible
Copy link

We also started seeing this a couple of days ago. Before that there was absolutely no problem. The lambda does not run in a VPC but is invoked by API Gateway

@code-xhyun
Copy link

code-xhyun commented Apr 16, 2021

I experience the same error When ts is compiled with js after putting axios in forEach, map
environment : Connect Lambda to SQS and run Lambda

2021-04-16T06:00:24.537Z 1f630065-c993-5f4e-bb70-5896e0bbfe96 INFO Error: Client network socket disconnected before secure TLS connection was established at connResetException (internal/errors.js:609:14) at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19) at Object.onceWrapper (events.js:420:28) at TLSSocket.emit (events.js:326:22) at endReadableNT (_stream_readable.js:1241:12) at processTicksAndRejections (internal/process/task_queues.js:84:21) { code: 'ECONNRESET', path: null, host: 'bot-api.kakao.com', port: 443, localAddress: undefined, }

@aagamdoshi
Copy link

Experiencing the same issue for AWS_Lambda while logging data into Sumologic, this is impacting us at a higher level. Requesting some ETA on the same

Client network socket disconnected before secure TLS connection was established at connResetException

Lambda issue

@hamiltonchua
Copy link

hamiltonchua commented Apr 27, 2021

+1 on this issue, we started experiencing this recently on a newly re-deployed micro service.
AWS Lambda on Nodejs 12 runtime

{ "errorType": "Runtime.UnhandledPromiseRejection", "errorMessage": "Error: Client network socket disconnected before secure TLS connection was established", "reason": { "message": "Client network socket disconnected before secure TLS connection was established", "name": "Error", "stack": "Error: Client network socket disconnected before secure TLS connection was established\n at connResetException (internal/errors.js:609:14)\n at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19)\n at Object.onceWrapper (events.js:420:28)\n at TLSSocket.emit (events.js:326:22)\n at TLSSocket.EventEmitter.emit (domain.js:483:12)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)" }

@salvadornava01
Copy link

image

Same issue on Lambda running nodejs 12

Is there anything we can do to solve it?

Best regards

@amitava82
Copy link

I've received the following response from the support. I'd request someone from AWS SDK team please address this.

Thank you for being patient while I was getting back with the response from the internal Lambda team.

The team has noted that being AWS SDK issue, they cannot comment on ETA. So their suggestion is to raise a separate GitHub Issue on AWS Javascript SDK repository (refer 1) to get direction attention of the AWS SDK developers and maintainers. 

Therefore I would request you to raise a GitHub Issue and share it with me so that I could contact the concerned team internally for quick response.

@kaydwithers
Copy link

We had this issue with our Nuxt.js application. An intermittent crash which happened around 5% of the time.

We were calling a function which used useRuntimeConfig() on created. Moving this function from created to mounted seems to have fixed it for us.

@dharmveer18
Copy link

I am getting same error

{ "errorType": "Runtime.UnhandledPromiseRejection", "errorMessage": "TimeoutError: read ECONNRESET", "reason": { "errorType": "TimeoutError", "errorMessage": "read ECONNRESET", "code": "TimeoutError", "message": "read ECONNRESET", "errno": -104, "syscall": "read", "hostname": "secretsmanager.amazonaws.com", "retryable": true, "stack": [ "Error: read ECONNRESET", " at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)", " at TLSWrap.callbackTrampoline (internal/async_hooks.js:130:17)" ] }, "promise": {}, "stack": [ "Runtime.UnhandledPromiseRejection: TimeoutError: read ECONNRESET", " at process. (/var/runtime/index.js:35:15)", " at process.emit (events.js:400:28)", " at process.emit (domain.js:475:12)", " at processPromiseRejections (internal/process/promises.js:245:33)", " at processTicksAndRejections (internal/process/task_queues.js:96:32)" ] }

@vilasjagani2000
Copy link

Today we are getting same error for Nodejs 12 lambdas
{
"Error": "TimeoutError",
"Cause": "{"errorType":"TimeoutError","errorMessage":"Client network socket disconnected before secure TLS connection was established","trace":["Error: Client network socket disconnected before secure TLS connection was established"," at connResetException (internal/errors.js:609:14)"," at TLSSocket.onConnectEnd (_tls_wrap.js:1557:19)"," at Object.onceWrapper (events.js:420:28)"," at TLSSocket.emit (events.js:326:22)"," at TLSSocket.EventEmitter.emit (domain.js:483:12)"," at endReadableNT (_stream_readable.js:1241:12)"," at processTicksAndRejections (internal/process/task_queues.js:84:21)"]}"
}

our aws-sdk version is 2.54.0

@kartheek-artium
Copy link

did you figure out the issue?

@Mektrode
Copy link

Mektrode commented Aug 2, 2022

FWIW we had this issue a few months ago and discovered it was due to an unhandled promise. We exported the function asynchronously and did not return a promise in any part of the subsequent code. What was weird however, was that the code worked fine until one day it didn't. It hadn't been touched in months and decided to just error out one day. Take from that what you will and double check to make sure you are handling the async flow of your lambdas with care. The sdk can behave quite strangely at times.

@kartheek-artium
Copy link

thanks a lot.

@fagiani
Copy link

fagiani commented Oct 3, 2022

@Mektrode I can confirm that!

FWIW we had this issue a few months ago and discovered it was due to an unhandled promise. We exported the function asynchronously and did not return a promise in any part of the subsequent code. What was weird however, was that the code worked fine until one day it didn't. It hadn't been touched in months and decided to just error out one day. Take from that what you will and double check to make sure you are handling the async flow of your lambdas with care. The sdk can behave quite strangely at times.

After finding an asynchronous call that did not respond with a proper promise, although it worked just fine locally and within an EC2 environment, lambda would just let it go and skip it's return and end the function execution early, which after understanding properly one of the lambda's goals I think it actually makes sense as its pricing includes execution time charges as a relevant part of the offer.

@ajredniwja it may not be actually related to the SDK but AWS could definitely do a better job here explaining clearly and providing some good examples with proposed refactors as it looks like a product feature and not a bug considering the explanation given above.

FWIW, I'd recommend anyone facing this issue to review the implementation and find where it could be improved promise wise in order to achieve its goal within lambda. For that, this stack overflow topic has been quite helpful to me as it explains both serialized and parallelized execution approaches!

Keep Rocking!

@klukiyan
Copy link

klukiyan commented Oct 15, 2022

We had this issue with our Nuxt.js application. An intermittent crash which happened around 5% of the time.

We were calling a function which used useRuntimeConfig() on created. Moving this function from created to mounted seems to have fixed it for us.

Hi @kaydwithers ,
I have this issue with NUXT SSR on Vercel. Can you please elaborate how you fixed it. It's unbearable that sometimes customers see 500 error instead of page.
Thank you

I replace the created() with mounted() and it seems that I don't have 500 errors anymore.
Thank you very much.
I can't believe I finally have a solution after digging for 3 days in a row.
Thank you

@klukiyan
Copy link

Still experiencing this with Nuxt on Vercel which is a shame because it's really fast. I'm not using Promise.all() anywhere in my code right now but it's possibly being used in the Nuxt core or in a package I'm using. I find that I usually hit the error when clicking a link that has metadata loaded by SSR.

Hi, @bradley-varol ,
did you find a solution with Nuxt?

@bradley-varol
Copy link

bradley-varol commented Oct 17, 2022

Still experiencing this with Nuxt on Vercel which is a shame because it's really fast. I'm not using Promise.all() anywhere in my code right now but it's possibly being used in the Nuxt core or in a package I'm using. I find that I usually hit the error when clicking a link that has metadata loaded by SSR.

Hi, @bradley-varol , did you find a solution with Nuxt?

Unfortunately not and with SSR enabled, my site regularly triggers a restart on a Digital Ocean App Platform app as well so I don't think my issue is Vercel/AWS specific. With 150+ components, it's very difficult to know where my issue lies, but since it's happening on all pages for me, it might be something to do with the nuxt-auth package or a globally loaded component. I will upgrade to Nuxt 3 soon and hopefully that solves the issue or at least helps me find it - I have disabled SSR in the meantime.

@klukiyan
Copy link

Still experiencing this with Nuxt on Vercel which is a shame because it's really fast. I'm not using Promise.all() anywhere in my code right now but it's possibly being used in the Nuxt core or in a package I'm using. I find that I usually hit the error when clicking a link that has metadata loaded by SSR.

Hi, @bradley-varol , did you find a solution with Nuxt?

Unfortunately not and with SSR enabled, my site regularly triggers a restart on a Digital Ocean App Platform app as well so I don't think my issue is Vercel/AWS specific. With 150+ components, it's very difficult to know where my issue lies, but since it's happening on all pages for me, it might be something to do with the nuxt-auth package or a globally loaded component. I will upgrade to Nuxt 3 soon and hopefully that solves the issue or at least helps me find it - I have disabled SSR in the meantime.
Hi @bradley-varol ,
I was so desperate that started rebuilding all the functionality for the SSG instead of SSR.
But then I tried replacing the created() events into mounted() and errors are gone. I don't know why it caused the issue, because it was returning 500s on pages that don't have the created(). buf for some reason it helped and all works very steadily.

@jony1993
Copy link

jony1993 commented Dec 9, 2022

We had the same issue yesterday for nuxt.js (ssr) and vercel.

Out of nowhere "Client network socket disconnected before secure TLS connection was established" error messages have been displayed for all our request. Never experienced it before and we also had no deployment/dependency update or something...Also started to work again without any change after some hours.

We are not using created() or Promise.all() at all.

@bradley-varol have you already found any solution or do you still have disabled SSR?

@bradley-varol
Copy link

bradley-varol commented Dec 9, 2022

We had the same issue yesterday for nuxt.js (ssr) and vercel.

Out of nowhere "Client network socket disconnected before secure TLS connection was established" error messages have been displayed for all our request. Never experienced it before and we also had no deployment/dependency update or something...Also started to work again without any change after some hours.

We are not using created() or Promise.all() at all.

@bradley-varol have you already found any solution or do you still have disabled SSR?

I have re-enabled SSR on Vercel, but I have wrapped large portions of my application in <client-only> tags. I have also made my Nuxt plugins client plugins where possible. There's an initial flash of content (e.g. when the Header mounts) but at least links now embed properly with OG meta.

I think with enough digging, I might be able to narrow the issue down to a particular plugin/package/component (maybe auth-next) but since I can't reproduce this locally and Nuxt 3 is now stable, I don't think i'll ever bother.

My issue wasn't intermitent - it was just not as noticeable during low periods of traffic since it seems to affect a percentage of requests. Perhaps you had increased traffic and that's why the issue seemed to suddenly increase in frequency.

@abandm
Copy link

abandm commented Jan 22, 2023

I have been looking into this as well for over a month and also realized it was because the lambda terminated execution after the response was sent, but before some of my scripts finished running. ( I was attempting to return a response early to quicken the response time for the user while leaving the lambda to continue updating the database for example because I thought it would stay active till the script finished.)

Many people may find this only occurs a small percentage of the time and the reason for it in this case was that the different parts of the code where finishing so close together that the script usually sent a response after all the code had run, but at times ran a little quicker and sent a response prior to the scripts finishing.

I have now changed all my scripts to run asynchronously with promises so all the code has completed running before sending a response to the client. This has solved my very frequent issue with this error.

I am unaware if this is a solution for all cases(if it were I would also agree with @fagiani that this is not actually a bug but a planned feature) but it was an absolute win for my case.

@leohku
Copy link

leohku commented Jan 26, 2023

I was attempting to return a response early to quicken the response time for the user while leaving the lambda to continue updating the database for example because I thought it would stay active till the script finished.

What a timely comment - I ran into this problem today and this was the key. Saved me from a lot of head banging. Thank you!

@RanVaknin RanVaknin added the p2 This is a standard priority issue label Mar 27, 2023
@dylanmrowe
Copy link

We're getting this error in prod daily trying to connect to DocumentDB. I've scoured our code, but we're properly awaiting all async calls. Node 18.x

It's possible our mongodb library is doing some async stuff behind the scenes that gets terminated when the lambda ends... We're trying to share pool of connections between all lambda invocations (as recommended by the mongodb docs). When the connection finally gets in a bad state it stays that way for at least an hour before resolving itself.

@mikaeltellhed
Copy link

I got the following error randomly on about every 100th invocation.

ERROR	Error: Client network socket disconnected before secure TLS connection was established
    at connResetException (node:internal/errors:692:14)
    at TLSSocket.onConnectEnd (node:_tls_wrap:1587:19)
    at TLSSocket.emit (node:events:539:35)
    at TLSSocket.emit (node:domain:475:12)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) ...

I noticed that I ONLY got the error once when a new Lambda instance was created, after the exception the runtime restarted and then there was no problem. This was in very old code that had worked just fine for a long time.

I found that I had a few initialisations of the AWS SDK object in the module scope:


const dynamodb = new AWS.DynamoDB();

function getProject(id, fn) {
  
  dynamodb.getItem({ ...

that I simply moved into the function:

function getProject(id, fn) {
  const dynamodb = new AWS.DynamoDB();
  dynamodb.getItem({ ...

That was the only change I made and now I have not seen an error for 12 hours.

@wasurocks
Copy link

I can confirm this issue is still happening for me.

I keep getting these randomly, after having awaited all my asynchronous functions everywhere.

Here is an example of the cloudwatch logs of one such event:

START RequestId: 1a176f55-1a53-4925-ac87-d4181642c2b3 Version: $LATEST
2023-05-24T08:37:25.958Z	5466903f-6186-4f9c-a79b-8fad08f10a9a	ERROR	Unhandled Promise Rejection 	{"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"Error: Client network socket disconnected before secure TLS connection was established","trace":["Runtime.UnhandledPromiseRejection: Error: Client network socket disconnected before secure TLS connection was established","    at process.<anonymous> (/var/runtime/index.js:35:15)","    at process.emit (events.js:400:28)","    at processPromiseRejections (internal/process/promises.js:245:33)","    at processTicksAndRejections (internal/process/task_queues.js:96:32)"]}
[ERROR] [1684917445978] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 400.
RequestId: 1a176f55-1a53-4925-ac87-d4181642c2b3 Error: Runtime exited with error: exit status 128
Runtime.ExitError
END RequestId: 1a176f55-1a53-4925-ac87-d4181642c2b3
REPORT RequestId: 1a176f55-1a53-4925-ac87-d4181642c2b3	Duration: 162.27 ms	Billed Duration: 163 ms	Memory Size: 1024 MB	Max Memory Used: 200 MB	

@rohitjpsingh
Copy link

2023-09-21T07:17:43.749Z 679ec5d9-5056-4d4f-b507-35daaa27047f INFO Something has gone wrong! Error: Client network socket disconnected before secure TLS connection was established
at connResetException (node:internal/errors:720:14)
at TLSSocket.onConnectEnd (node:_tls_wrap:1605:19)
at TLSSocket.emit (node:events:526:35)
at endReadableNT (node:internal/streams/readable:1359:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
code: 'ECONNRESET',
path: null,
host: 'fcm.googleapis.com',
port: 443,
localAddress: undefined
}

some time it's working and notification sent but some time it's error displaying above error. Any help?

@jakob-lilliemarck
Copy link

jakob-lilliemarck commented Jan 18, 2024

We came across this issue and it turned out to be due to a missing await for an outbound http request using axios.get(). The lambda returned before the request had finished, so the connection was then closed before the call to axios.get() had terminated 💡 Make sure you got no pending requests or promises at the time of returning from the lambda

@zshzbh
Copy link

zshzbh commented Aug 28, 2024

Hello there,

Sorry for the long wait, this is what the service team suggests -

Function Details

arn:aws:lambda:us-east-1:971156808745:function:cch-ea-web-lmda-platform-graphql-mobile-dev
Nodejs16.x
timeout 30 seconds
memory 128 MB
Function with VPC/Subnet/SG settings. Networking seems correct
Lambda is invoked by an API GW

"errorMessage\":\"Unable to generate certificate due to \\nError: Client network socket disconnected before secure TLS connection was established\

The above shows that the endpoint sent a reset for the connection. Further the nodejs documentation explains this further:

"ECONNRESET (Connection reset by peer): A connection was forcibly closed by a peer. This normally results from a loss of the connection on the remote socket due to a timeout or reboot. Commonly encountered via the http and net modules."

~ https://nodejs.org/api/errors.html

If you are not already using retry logic and back-off, then you should consider it, however rather than just retry the connection they should attempt to re-instantiate the client. This will force a DNS resolution again, this is particularly helpful if in the case the endpoint has multiple IPs to connect to (and one is problematic at the time)

Another common issue is inefficient connection reuse, if already not considered by the customer. Recommended practice is to establish connections in the init of the function execution, then reuse connections (HTTP Keepalive). Then in the handler the customer will need to use try/catch logic to reconnect in the event of an error.

According to the comments under this thread, using await for an outbound http request and make sure the connection is closed after receiving the response.

from this github issue I found the following harbinger/cases:
#3591
nodejs/node#3692
#862
https://harbinger.amazon.com/notices/44862
https://paragon-na.amazon.com/hz/view-case?caseId=7831443621#/

Additionally, I see the link https://stackoverflow.com/questions/56293822/how-to-fix-eproto-error-after-upgrading-nodes-version . Please advise customer to test with the solution suggested.

Other related links that could be helpful:
https://stackoverflow.com/questions/34411314/aws-dynamodb-gives-write-eproto/35251621

Hope this helps!!

@zshzbh zshzbh closed this as completed Aug 28, 2024
@zshzbh zshzbh self-assigned this Aug 28, 2024
@zshzbh
Copy link

zshzbh commented Aug 28, 2024

AWS SDK for JavaScript v2 will enter maintenance mode on September 8, 2024 and reach end-of-support on September 8, 2022.

AWS will limit SDK releases to address critical bug fixes and security issues only. The SDK will not receive API updates for new or existing services, or be updated to support new regions.

We recommend that you update to AWS SDK for JavaScript v3. AWS SDK for JavaScript v3 provides improved features, enhanced performance, and continued support from AWS.

Please refer to this announcement for more information.

I'm closing this issue, please let us know if there's any other questions!

Thanks!
Maggie

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p2 This is a standard priority issue service-api This issue is due to a problem in a service API, not the SDK implementation.
Projects
None yet
Development

No branches or pull requests