-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AWS-Lambda] - NetworkingError: Client network socket disconnected before secure TLS connection was established #3591
Comments
I got something very similar: intermittent, similar error, different service (S3):
|
It seems to be a problem at Lambda. Our support from aws: |
@robsonkades, thanks for providing the update, can you share the ticket number that you have with AWS and I can look into it, would like to close the issue if the issue is not related to the SDK. |
@ajredniwja, Thank you for your help! Case 7831443621 |
@robsonkades, thanks for providing that, I can reach out to the team related to that and can update once I hear back from them. |
Hi - we're also seeing some instances of this in a NodeJs lambda, are there any updates? |
@rcoundon No solution so far |
Ok, thanks for the update. Any ETA? |
I was not able to get any updates as of yet, I have contacted the engineers associated and will update you guys once I have some more insight from Lambda team. |
I face same error in aws lambda, I am using nodejs |
In case it's helpful, our issue is related to writes to S3. |
Same for us. Seems to affect quite significantly our Lambdas. Thanks @ajredniwja for the update. |
We've also got the same issue with AppSync - happening on a somewhat sensitive service. Any idea how we can mitigate this until there is a proper fix? |
We had the same issue reading from |
@brucemackenzie the lambda functions I have do not run in a VPC so I doubt this is a VPC specific issue. The primary function where this is causing us a concern has the issue does run on a cron - every minute - so not sure if that has any impact. However, I am possibly more concerned should this could happen on webhook requests. Especially as I may not be guaranteed a retry by a third-party service should the function fail. |
We don't use VPC |
Seeing same issue appearing here. |
+1 same issue. |
+1 same issue while calling a 3rd party service via https
|
+1 same issue
|
@illia-sh yes, |
We also started seeing this a couple of days ago. Before that there was absolutely no problem. The lambda does not run in a VPC but is invoked by API Gateway |
I experience the same error When ts is compiled with js after putting axios in forEach, map
|
+1 on this issue, we started experiencing this recently on a newly re-deployed micro service.
|
I've received the following response from the support. I'd request someone from AWS SDK team please address this.
|
We had this issue with our Nuxt.js application. An intermittent crash which happened around 5% of the time. We were calling a function which used |
I am getting same error { "errorType": "Runtime.UnhandledPromiseRejection", "errorMessage": "TimeoutError: read ECONNRESET", "reason": { "errorType": "TimeoutError", "errorMessage": "read ECONNRESET", "code": "TimeoutError", "message": "read ECONNRESET", "errno": -104, "syscall": "read", "hostname": "secretsmanager.amazonaws.com", "retryable": true, "stack": [ "Error: read ECONNRESET", " at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)", " at TLSWrap.callbackTrampoline (internal/async_hooks.js:130:17)" ] }, "promise": {}, "stack": [ "Runtime.UnhandledPromiseRejection: TimeoutError: read ECONNRESET", " at process. (/var/runtime/index.js:35:15)", " at process.emit (events.js:400:28)", " at process.emit (domain.js:475:12)", " at processPromiseRejections (internal/process/promises.js:245:33)", " at processTicksAndRejections (internal/process/task_queues.js:96:32)" ] } |
Today we are getting same error for Nodejs 12 lambdas our aws-sdk version is 2.54.0 |
did you figure out the issue? |
FWIW we had this issue a few months ago and discovered it was due to an unhandled promise. We exported the function asynchronously and did not return a promise in any part of the subsequent code. What was weird however, was that the code worked fine until one day it didn't. It hadn't been touched in months and decided to just error out one day. Take from that what you will and double check to make sure you are handling the async flow of your lambdas with care. The sdk can behave quite strangely at times. |
thanks a lot. |
@Mektrode I can confirm that!
After finding an asynchronous call that did not respond with a proper promise, although it worked just fine locally and within an EC2 environment, lambda would just let it go and skip it's return and end the function execution early, which after understanding properly one of the lambda's goals I think it actually makes sense as its pricing includes execution time charges as a relevant part of the offer. @ajredniwja it may not be actually related to the SDK but AWS could definitely do a better job here explaining clearly and providing some good examples with proposed refactors as it looks like a product feature and not a bug considering the explanation given above. FWIW, I'd recommend anyone facing this issue to review the implementation and find where it could be improved promise wise in order to achieve its goal within lambda. For that, this stack overflow topic has been quite helpful to me as it explains both serialized and parallelized execution approaches! Keep Rocking! |
Hi @kaydwithers , I replace the created() with mounted() and it seems that I don't have 500 errors anymore. |
Hi, @bradley-varol , |
Unfortunately not and with SSR enabled, my site regularly triggers a restart on a Digital Ocean App Platform app as well so I don't think my issue is Vercel/AWS specific. With 150+ components, it's very difficult to know where my issue lies, but since it's happening on all pages for me, it might be something to do with the nuxt-auth package or a globally loaded component. I will upgrade to Nuxt 3 soon and hopefully that solves the issue or at least helps me find it - I have disabled SSR in the meantime. |
|
We had the same issue yesterday for nuxt.js (ssr) and vercel. Out of nowhere "Client network socket disconnected before secure TLS connection was established" error messages have been displayed for all our request. Never experienced it before and we also had no deployment/dependency update or something...Also started to work again without any change after some hours. We are not using @bradley-varol have you already found any solution or do you still have disabled SSR? |
I have re-enabled SSR on Vercel, but I have wrapped large portions of my application in I think with enough digging, I might be able to narrow the issue down to a particular plugin/package/component (maybe My issue wasn't intermitent - it was just not as noticeable during low periods of traffic since it seems to affect a percentage of requests. Perhaps you had increased traffic and that's why the issue seemed to suddenly increase in frequency. |
I have been looking into this as well for over a month and also realized it was because the lambda terminated execution after the response was sent, but before some of my scripts finished running. ( I was attempting to return a response early to quicken the response time for the user while leaving the lambda to continue updating the database for example because I thought it would stay active till the script finished.) Many people may find this only occurs a small percentage of the time and the reason for it in this case was that the different parts of the code where finishing so close together that the script usually sent a response after all the code had run, but at times ran a little quicker and sent a response prior to the scripts finishing. I have now changed all my scripts to run asynchronously with promises so all the code has completed running before sending a response to the client. This has solved my very frequent issue with this error. I am unaware if this is a solution for all cases(if it were I would also agree with @fagiani that this is not actually a bug but a planned feature) but it was an absolute win for my case. |
What a timely comment - I ran into this problem today and this was the key. Saved me from a lot of head banging. Thank you! |
We're getting this error in prod daily trying to connect to DocumentDB. I've scoured our code, but we're properly awaiting all async calls. Node 18.x It's possible our mongodb library is doing some async stuff behind the scenes that gets terminated when the lambda ends... We're trying to share pool of connections between all lambda invocations (as recommended by the mongodb docs). When the connection finally gets in a bad state it stays that way for at least an hour before resolving itself. |
I got the following error randomly on about every 100th invocation.
I noticed that I ONLY got the error once when a new Lambda instance was created, after the exception the runtime restarted and then there was no problem. This was in very old code that had worked just fine for a long time. I found that I had a few initialisations of the AWS SDK object in the module scope:
that I simply moved into the function:
That was the only change I made and now I have not seen an error for 12 hours. |
I can confirm this issue is still happening for me. I keep getting these randomly, after having awaited all my asynchronous functions everywhere. Here is an example of the cloudwatch logs of one such event:
|
2023-09-21T07:17:43.749Z 679ec5d9-5056-4d4f-b507-35daaa27047f INFO Something has gone wrong! Error: Client network socket disconnected before secure TLS connection was established some time it's working and notification sent but some time it's error displaying above error. Any help? |
We came across this issue and it turned out to be due to a missing |
Hello there, Sorry for the long wait, this is what the service team suggests - Function Details
The above shows that the endpoint sent a reset for the connection. Further the nodejs documentation explains this further:
~ https://nodejs.org/api/errors.html If you are not already using retry logic and back-off, then you should consider it, however rather than just retry the connection they should attempt to re-instantiate the client. This will force a DNS resolution again, this is particularly helpful if in the case the endpoint has multiple IPs to connect to (and one is problematic at the time) Another common issue is inefficient connection reuse, if already not considered by the customer. Recommended practice is to establish connections in the init of the function execution, then reuse connections (HTTP Keepalive). Then in the handler the customer will need to use try/catch logic to reconnect in the event of an error. According to the comments under this thread, using from this github issue I found the following harbinger/cases: Additionally, I see the link https://stackoverflow.com/questions/56293822/how-to-fix-eproto-error-after-upgrading-nodes-version . Please advise customer to test with the solution suggested. Other related links that could be helpful: Hope this helps!! |
AWS SDK for JavaScript v2 will enter maintenance mode on September 8, 2024 and reach end-of-support on September 8, 2022. AWS will limit SDK releases to address critical bug fixes and security issues only. The SDK will not receive API updates for new or existing services, or be updated to support new regions. We recommend that you update to AWS SDK for JavaScript v3. AWS SDK for JavaScript v3 provides improved features, enhanced performance, and continued support from AWS. Please refer to this announcement for more information. I'm closing this issue, please let us know if there's any other questions! Thanks! |
We are having problems with several of our services to access DynamoDb. This problem is intermittent, but occurs randomly several times a day at each of the services on (AWS LAMBDA).
Any suggestion?
The text was updated successfully, but these errors were encountered: