Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ReadEntityStateAsync sometimes fails with HTTP 412 when calling Azure Storage #1406

Closed
jriouxrandstadca opened this issue Jul 7, 2020 · 5 comments · Fixed by Azure/durabletask#416
Closed

ReadEntityStateAsync sometimes fails with HTTP 412 when calling Azure Storage #1406

jriouxrandstadca opened this issue Jul 7, 2020 · 5 comments · Fixed by Azure/durabletask#416
Assignees
@bachuv
Labels
bug dtfx
Milestone
Extension v2.3.0

Comments

@jriouxrandstadca
Copy link

jriouxrandstadca commented Jul 7, 2020 •
edited
Loading

Description

We have a function app that's using durable entities to store a big chunk of data. The app can run seamlessly for days and then suddenly we have a wave of StorageException.

Message: Exception while executing function: One or more errors occurred. (The condition specified using HTTP conditional header(s) is not met.) The condition specified using HTTP conditional header(s) is not met.
Exception type: Microsoft.WindowsAzure.Storage.StorageException
Failed method: DurableTask.AzureStorage.MessageManager.Decompress

Known workarounds

None that we know of.

App Details

  • Durable Functions extension version (e.g. v1.8.3): 2.2.2
  • Azure Functions runtime version (1.0 or 2.0): 3.0
  • Programming language used: c#

Screenshots

image

If deployed to Azure

  • Timeframe issue observed: approx. 2020-07-07T17:22:00.000Z to 2020-07-07T19:18:00.000Z
  • Function name(s): OtsTimesheetAdded
  • Azure region: Canada Central
  • Orchestration instance ID(s):
@ghost ghost added the Needs: Triage 🔍 label Jul 7, 2020
@cgillum
Copy link
Member

cgillum commented Jul 7, 2020

Thanks for these details. It seems to me that the size of your entity is sufficiently large that we're saving it into blob storage, and somehow the download of that state is somehow failing (relevant source code is here). I'm not sure why it would be failing though.

Just to confirm, you're seeing function execution failures because of these errors?

@cgillum cgillum added dtfx Needs: Author Feedback Waiting for the author of the issue to respond to a question and removed Needs: Triage 🔍 labels Jul 7, 2020
@jriouxrandstadca
Copy link
Author

That's right. We have an EventGridTrigger function that gets triggered and that function calls ReadEntityStateAsync to get state of the entity. That call fails and so the EventGridTrigger fails and the event gets deadletter by EventGrid after multiple retries.

@ghost ghost added Needs: Attention 👋 and removed Needs: Author Feedback Waiting for the author of the issue to respond to a question labels Jul 7, 2020
@cgillum cgillum added Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. and removed Needs: Attention 👋 labels Jul 7, 2020
@cgillum cgillum added this to the v2.3.0 milestone Jul 7, 2020
@bachuv
Copy link
Collaborator

bachuv commented Jul 10, 2020

@jriouxrandstadca I wasn't able to reproduce the exception in order to find a fix. Is there any other information you can give to help me reproduce it?

@jriouxrandstadca
Copy link
Author

jriouxrandstadca commented Jul 13, 2020 •
edited
Loading

@bachuv Hi, it happened again this morning. Http error 412 when trying to read a blob history-0000000000000001-ExecutionStarted-Input.json.gz from the large messages container. Here is a part of the call stack

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__19.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.17.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.17.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.17.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 325)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsyncCore>d__16.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.17.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.17.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.17.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 117)
Inner exception System.AggregateException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at System.Threading.Tasks.Task.ThrowIfExceptional (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.Task`1.GetResultCore (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.Task`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.WindowsAzure.Storage.Blob.BlobReadStream+<>c__DisplayClass4_0.<Read>b__0 (Microsoft.WindowsAzure.Storage, Version=9.3.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.RunWithoutSynchronizationContext (Microsoft.WindowsAzure.Storage, Version=9.3.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.IO.Compression.DeflateStream.ReadCore (System.IO.Compression, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.IO.Compression.DeflateStream.Read (System.IO.Compression, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at DurableTask.AzureStorage.MessageManager.Decompress (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\MessageManager.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 229)
   at DurableTask.AzureStorage.MessageManager+<DownloadAndDecompressAsBytesAsync>d__18.MoveNext (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\MessageManager.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 210)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.AzureStorage.MessageManager+<FetchLargeMessageIfNecessary>d__12.MoveNext (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\MessageManager.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 124)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore+<ConvertFromAsync>d__41.MoveNext (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\Tracking\AzureTableTrackingStore.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 562)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore+<FetchInstanceStatusInternalAsync>d__40.MoveNext (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\Tracking\AzureTableTrackingStore.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 533)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore+<GetStateAsync>d__38.MoveNext (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\Tracking\AzureTableTrackingStore.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 484)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore+<GetStateAsync>d__37.MoveNext (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\Tracking\AzureTableTrackingStore.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 478)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.AzureStorage.AzureStorageOrchestrationService+<GetOrchestrationStateAsync>d__96.MoveNext (DurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.AzureStorage\AzureStorageOrchestrationService.csDurableTask.AzureStorage, Version=1.7.7.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 1359)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.AzureStorageDurabilityProvider+<RetrieveSerializedEntityState>d__15.MoveNext (Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: d:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\AzureStorageDurabilityProvider.csMicrosoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: 86)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient+<ReadEntityStateAsync>d__42`1.MoveNext (Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: d:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableClient.csMicrosoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: 457)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

Also some custom properties of the failed call if it helps
image

@cgillum
Copy link
Member

cgillum commented Jul 13, 2020

@bachuv I found this post that describes a similar problem: https://stackoverflow.com/questions/5094089/azure-blob-the-condition-specified-using-http-conditional-headers-is-not-met. Basically, it looks like certain blob APIs can suffer from race conditions if someone tries to download the blob while it is being actively updated (which is quite surprising to me).

Looking at our code, it seems like we might be at risk for the exact same race condition:
https://github.com/Azure/durabletask/blob/15345fb9db38c5c74c23cb22e8349c3a3d165a28/src/DurableTask.AzureStorage/MessageManager.cs#L209

The solution in the post implies that we should use cloudBlockBlob.DownloadToStreamAsync(Stream) instead of cloudBlockBlob.OpenReadAsync() to avoid these kinds of issues. This might actually improve our performance too since the latter operation appears to be doing multiple storage requests under the covers!

@cgillum cgillum added bug and removed Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. labels Jul 13, 2020
@bachuv bachuv mentioned this issue Jul 13, 2020
Merged
@cgillum cgillum changed the title Weird StorageException out of nowhere ReadEntityStateAsync sometimes fails with HTTP 412 when calling Azure Storage Jul 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bachuv bachuv

Labels
bug dtfx
Projects
None yet
Milestone
Extension v2.3.0
Development

Successfully merging a pull request may close this issue.

Use DownloadToStreamAsync(Stream) to download blob contents Azure/durabletask
3 participants
@cgillum @bachuv @jriouxrandstadca