Authentication error when executing HTTP trigger #1490

charisk · 2020-09-22T15:05:53Z

Description

Occasionally, when running an HTTP trigger function, an authentication error occurs.

Expected behavior

No authentication errors happening when the HTTP trigger is executing.

Actual behavior

Every now and then, we seem to get an authentication error when executing an HTTP trigger in our function app. The HTTP trigger kicks off an orchestration. The following error is logged.

Message:

Exception while executing function: Functions.XXXXX Result: Failure
Exception: Error: The operation failed with an unexpected status code: 401. Details: ""
Stack: Error: The operation failed with an unexpected status code: 401. Details: ""
    at DurableOrchestrationClient.createGenericError (D:\home\site\wwwroot\node_modules\durable-functions\lib\src\durableorchestrationclient.js:591:16)
    at DurableOrchestrationClient.<anonymous> (D:\home\site\wwwroot\node_modules\durable-functions\lib\src\durableorchestrationclient.js:402:44)
    at Generator.next (<anonymous>)
    at fulfilled (D:\home\site\wwwroot\node_modules\durable-functions\lib\src\durableorchestrationclient.js:5:58)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Call Stack:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__21.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 338)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsyncCore>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 123)
Inner exception Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__8.MoveNext (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script\Description\Workers\WorkerFunctionInvoker.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 85)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script\Description\FunctionInvokerBase.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 85)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator+<Coerce>d__3`1.MoveNext (Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script\Description\FunctionGenerator.csMicrosoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 225)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeAsync>d__29.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 589)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__28.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 537)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__27.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 481)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__21.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 290)

The function app is using app key authentication (using the x-functions-key header).

This might be relevant to #1348 although I believe the extension bundle version that we're running is 1.8.5 as this is what I see loaded in the KUDU process explorer (Microsoft.Azure.WebJobs.Extensions.DurableTask.dll 1.8.5)

Relevant source code snippets

N/A.

Known workarounds

N/A

App Details

Durable Functions extension version (e.g. v1.8.3): 1.8.5
Azure Functions runtime version (1.0 or 2.0): 2.0
Programming language used: TypeScript

Screenshots

N /A

If deployed to Azure

Timeframe issue observed: It has happened 9 times in the last 5 days.
Azure region: West Europe

The text was updated successfully, but these errors were encountered:

cgillum · 2020-09-22T20:13:53Z

@charisk Which version of the durable-functions npm package are you using? If it's an old version, then that could explain why you're still running into this. Also, you mentioned you see this occasionally...does that mean sometimes it works fine and only sometimes it fails with a 401?

Adding @davidmrdavid

charisk · 2020-09-22T20:20:21Z

@cgillum we're using version 1.4.3 of durable-functions. And yes, it works fine most times but every now and then we see this error, even though the function is always called the same way.

cgillum · 2020-09-22T20:40:26Z

@charisk okay, that's very odd. I think we'll need to look at our internal telemetry to understand where that 401 is coming from. Is there any more information you can provide us that will help us identify your function app, like a function name, the app name, or a function invocation ID?

charisk · 2020-09-22T20:47:42Z

Thanks for looking into this @cgillum!

I see an invocation ID of 67c18782-3271-4d4b-8ac0-0ffe2494c715 in App Insights - is that what you're after? I'm happy to provide the rest of the info somewhere less public.

charisk · 2020-09-24T10:18:35Z

Hey @cgillum, did you have a chance to look into the issue by any chance?

cgillum · 2020-09-24T17:35:47Z

Hey @charisk. I looked at it just now, and I confirmed that it looks like your app is indeed not using the local RPC endpoint, which partly explains why you got that 401. I also confirmed that your app appears to have been using v1.8.5 of the extension. According to our release notes for v1.8.5, this should not be the case.

You may want to consider upgrading to a newer version of the extension. I know we made some improvements in v1.8.6, though I'm not sure whether they would specifically help in this case.

@ConnorMcMahon can you help investigate this issue (I'm currently on-call for App Service and weighed down by other issues at the moment)?

ConnorMcMahon · 2020-09-24T23:30:19Z

@charisk,

I cannot for the life of me figure out why your app is not using the local RPC endpoint. Given your host.json and the value for FUNCTIONS_WORKER_RUNTIME, it should be turned on by default, but it isn't...

In the meantime, I would try explicitly setting it to true in your host.json. You can do that by adding this to your host.json:

 "extensions": {
   "durableTask": {
      "localRpcEndpointEnabled": true
    }
  }

charisk · 2020-09-27T18:48:37Z

Thank you @cgillum and @ConnorMcMahon for looking into the issue for us.

In the meantime, I would try explicitly setting it to true in your host.json. You can do that by adding this to your host.json:
 "extensions": {
   "durableTask": {
      "localRpcEndpointEnabled": true
    }
  }

We've turned localRpcEndpointEnabled to false due to this bug, which if I understand right is fixed in version 1.8.6.

We've upgraded to 1.8.6 (by executing func extensions install --package Microsoft.Azure.WebJobs.Extensions.DurableTask --version 1.8.6) and will be doing some testing to see if the issue still occurs. I'll post an update in a couple of days.

ConnorMcMahon · 2020-09-29T23:23:43Z

@charisk, glad you were able to figure out how to install 1.8.6, installing extensions can be rather tricky for non-.NET language.

As a long-term solution, I would recommend transitioning from v1 of the extension bundles to v2. We are unlikely to release many new versions of v1, just due to the maintenance cost of adding to an old branch like that, so most future bug fixes and improvements will be best gotten through v2 of the extension bundle. There are instructions for that here.

There is no rush if v1.8.6 fixes your issue (which it should given the description), but for future proofing your app that would be my recommendation!

ghost · 2020-10-04T00:00:07Z

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

charisk · 2020-10-06T08:13:35Z

We've now had the fix live for a few days and have not seen occurrences of this issue so the bug can be closed.

Thanks for the help @cgillum and @ConnorMcMahon!

ghost added the Needs: Triage 🔍 label Sep 22, 2020

cgillum added Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. and removed Needs: Triage 🔍 labels Sep 22, 2020

ConnorMcMahon added Needs: Author Feedback Waiting for the author of the issue to respond to a question and removed Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. labels Sep 24, 2020

ghost added Needs: Attention 👋 and removed Needs: Author Feedback Waiting for the author of the issue to respond to a question labels Sep 27, 2020

ConnorMcMahon added Needs: Author Feedback Waiting for the author of the issue to respond to a question and removed Needs: Attention 👋 labels Sep 29, 2020

ghost added the no-recent-activity label Oct 4, 2020

ghost added Needs: Attention 👋 and removed Needs: Author Feedback Waiting for the author of the issue to respond to a question no-recent-activity labels Oct 6, 2020

ConnorMcMahon closed this as completed Oct 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Authentication error when executing HTTP trigger #1490

Authentication error when executing HTTP trigger #1490

charisk commented Sep 22, 2020 •

edited

Loading

cgillum commented Sep 22, 2020

charisk commented Sep 22, 2020

cgillum commented Sep 22, 2020

charisk commented Sep 22, 2020

charisk commented Sep 24, 2020

cgillum commented Sep 24, 2020

ConnorMcMahon commented Sep 24, 2020

charisk commented Sep 27, 2020 •

edited

Loading

ConnorMcMahon commented Sep 29, 2020 •

edited

Loading

ghost commented Oct 4, 2020

charisk commented Oct 6, 2020 •

edited

Loading

Authentication error when executing HTTP trigger #1490

Authentication error when executing HTTP trigger #1490

Comments

charisk commented Sep 22, 2020 • edited Loading

Description

Expected behavior

Actual behavior

Relevant source code snippets

Known workarounds

App Details

Screenshots

If deployed to Azure

cgillum commented Sep 22, 2020

charisk commented Sep 22, 2020

cgillum commented Sep 22, 2020

charisk commented Sep 22, 2020

charisk commented Sep 24, 2020

cgillum commented Sep 24, 2020

ConnorMcMahon commented Sep 24, 2020

charisk commented Sep 27, 2020 • edited Loading

ConnorMcMahon commented Sep 29, 2020 • edited Loading

ghost commented Oct 4, 2020

charisk commented Oct 6, 2020 • edited Loading

charisk commented Sep 22, 2020 •

edited

Loading

charisk commented Sep 27, 2020 •

edited

Loading

ConnorMcMahon commented Sep 29, 2020 •

edited

Loading

charisk commented Oct 6, 2020 •

edited

Loading