getting Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException in some cases when the underlying machine starts up #597

tonakai · 2022-06-16T07:06:54Z

Original ticket was created here: it is suggested that i should ask here for more details: Azure/azure-functions-host#8438

Investigative information

Please provide the following:

Timestamp: 2022-06-07T00:11:47.881Z (i've more timestamps but this one is the most recent.)
Function App version: 4.3.2.18186
Function App name:
Function name(s) (as appropriate):
Invocation ID: 3c07d012-84f0-4437-994c-0b5079e32214
Region: Central US

Repro steps

Provide the steps required to reproduce the problem:
i don't have exact steps to reproduce this issue, however it seems to happen during startup of each instance.

Expected behavior

Provide a description of the expected behavior.

no error

Actual behavior

Provide a description of the actual behavior observed.

i see an exception with the following call stack:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:353)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.3.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:96)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.3.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs:82)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator+<Coerce>d__3`1.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.3.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script/Description/FunctionGenerator.cs:225)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:582)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:528)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.33.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:307)

they all seem to have similar pattern like this one:

I also couldn't see any errors being reported using the context since it looks like it happens before my function runs.

Known workarounds

this error happens just once per machine, we have some try logic that handles it but ideally we shouldn't see any errors.
we have about 80k requests and 10 of them failed like this in the last 24 hours.

Related information

Provide any related information

This is a typescript/node.js project
HttpBinding

The text was updated successfully, but these errors were encountered:

ejizba · 2022-06-17T20:51:44Z

Sarah mentioned this error node exited with code -1073740791 and that error code converted to hex is 0xC0000409. Based on some googling, seems like that code means STATUS_STACK_BUFFER_OVERRUN or STATUS_FAIL_FAST_EXCEPTION

This is likely caused by something you're doing in your code, as opposed to something on our side. If your code is open source or you're comfortable sharing it here, we can take a look and see if we notice anything.

ronkot · 2022-06-20T08:30:36Z

We also encountered the node exited with code -1073740791 error on 2022-06-17T09:28:26.3823561Z. This was (probably) the first time we saw this error and it occurred without any changes to our code.

I'm not sure if the error is related to our own NodeJS code or the underlying Azure infra, but here some possible places in our code that could be related to the problem:

Running deflateSync from zlib module
Uploading data to blob using @azure/storage-blob
Writing data to queue output binding

Unfortunately we don't have comprehensive logging so targeting the place exactly is not possible.

We're continuing investigations and will report back if we find something. Also, we can provide more information if it helps your investigation.

ejizba · 2022-06-21T17:43:20Z

Yeah this kind of error (where node exits quickly) is hard because you don't always get helpful stack traces and whatnot. If you can add more comprehensive logging I'm sure that would help. If the issue is happening consistently, another helpful option is to try to create a sample app with the minimum amount of code needed to reproduce.

As an aside, @hossam-nasr mentioned he has seen this error before when people use axios. Are you using axios by chance?

ronkot · 2022-06-23T06:00:14Z

I think I can't build a sample app as this problem occurs very rarely, once in a few months or so. We'll need to add more logging and follow this issue for more clues.

Yes, we're using axios in this function app, but not in the failing function. So, the deployment zip package does contain axios, but it's not necessarily loaded to memory in this function.. 🤔

hossam-nasr · 2022-06-23T17:20:56Z

@ronkot There's a good chance axios might be what's causing your app to crash. What version of axios and node are you using? For some versions, axios is known to cause node to crash with this error code. See axios issue here: axios/axios#3396 and node issue here: nodejs/node#36325

ronkot · 2022-06-23T18:18:09Z

@hossam-nasr thanks for pointers, those look quite interesting indeed! I just started my 3 week summer holidays but will reach back after that!

tonakai · 2022-07-04T07:43:56Z

so we are running on node.js 16 LTS and we do have axios dependency "axios": "^0.21.1", not directly but it looks like couple of azure libraries using it. I'll take a look if they updated.

ronkot · 2022-07-28T07:11:31Z

@hossam-nasr thanks for pointers, those look quite interesting indeed! I just started my 3 week summer holidays but will reach back after that!

Hi again 😎 Just checked our error logs from past 4 months and this problem have occurred only once. The problem is so rare that we won't put more effort to find out the root cause. Thanks a lot for the kind advice, I'll come back if the issue pops up again 🙏

hossam-nasr · 2022-08-29T19:13:43Z

@tonakai Did you check if the axios dependency was updated? Are you still seeing this error?

tonakai · 2022-08-30T07:34:43Z

@hossam-nasr sorry i was a bit sidetracked i looked at the last 7 days and it is still there 4 out of 500k requests failed.
one of those reported:
Exception while executing function:<function-name> node exited with code 1 (0x1) LanguageWorkerConsoleLog[error] Worker e7f07bc8-5269-472b-9601-b179deef3a94 uncaught exception (learn more: https://go.microsoft.com/fwlink/?linkid=2097909 ): Error: Connection terminated unexpectedly at Connection.<anonymous> (D:\home\site\wwwroot\node_modules\pg\lib\client.js:132:73) at Object.onceWrapper (node:events:641:28) at Connection.emit (node:events:527:28) at Socket.<anonymous> (D:\home\site\wwwroot\node_modules\pg\lib\connection.js:57:12) at Socket.emit (node:events:539:35) at TCP.<anonymous> (node:net:709:12)

but not sure if it is somehow related since this one mentioned node-pg library, but we didn't observe this error before as well.

ejizba · 2022-09-12T17:59:22Z

We believe the root cause here was the axios dependency as @hossam-nasr mentioned. We have a separate work item to help users solves these kinds of issues much faster. Closing in favor of #593

ghost added the Needs: Triage (Functions) label Jun 16, 2022

ejizba added bug investigate and removed Needs: Triage (Functions) labels Jun 21, 2022

ejizba added this to the Backlog Candidates milestone Jun 21, 2022

ejizba closed this as completed Sep 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

getting Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException in some cases when the underlying machine starts up #597

getting Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException in some cases when the underlying machine starts up #597

tonakai commented Jun 16, 2022

ejizba commented Jun 17, 2022

ronkot commented Jun 20, 2022 •

edited

Loading

ejizba commented Jun 21, 2022

ronkot commented Jun 23, 2022

hossam-nasr commented Jun 23, 2022 •

edited

Loading

ronkot commented Jun 23, 2022

tonakai commented Jul 4, 2022

ronkot commented Jul 28, 2022

hossam-nasr commented Aug 29, 2022

tonakai commented Aug 30, 2022

ejizba commented Sep 12, 2022

getting Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException in some cases when the underlying machine starts up #597

getting Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException in some cases when the underlying machine starts up #597

Comments

tonakai commented Jun 16, 2022

Investigative information

Repro steps

Expected behavior

Actual behavior

Known workarounds

Related information

ejizba commented Jun 17, 2022

ronkot commented Jun 20, 2022 • edited Loading

ejizba commented Jun 21, 2022

ronkot commented Jun 23, 2022

hossam-nasr commented Jun 23, 2022 • edited Loading

ronkot commented Jun 23, 2022

tonakai commented Jul 4, 2022

ronkot commented Jul 28, 2022

hossam-nasr commented Aug 29, 2022

tonakai commented Aug 30, 2022

ejizba commented Sep 12, 2022

ronkot commented Jun 20, 2022 •

edited

Loading

hossam-nasr commented Jun 23, 2022 •

edited

Loading