Crash using medium test case #263

ericleigh007 · 2023-04-28T18:52:36Z

New test case testing load with 20 orchestrators with 20 activities each encountered a crash:

2023-04-28T18:38:51Z   [Error]   Part03 !!! Encountered exception while working on store in StoreWorker.Process: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Xml.XmlStreamNodeWriter.FlushBuffer()
   at System.Xml.XmlStreamNodeWriter.UnsafeWriteUTF8Chars(Char* chars, Int32 charCount)
   at System.Xml.XmlUTF8NodeWriter.UnsafeWriteEscapedText(Char* chars, Int32 count)
   at System.Xml.XmlUTF8NodeWriter.WriteEscapedText(String s)
   at System.Xml.XmlBaseWriter.WriteString(String value)
   at System.Xml.XmlBaseWriter.WriteValue(String value)
   at System.Xml.XmlDictionaryAsyncCheckWriter.WriteValue(String value)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteString(XmlWriterDelegator xmlWriter, String value, XmlDictionaryString name, XmlDictionaryString ns)
   at WriteTaskCompletedEventToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , ClassDataContract )
   at System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithXsiType(XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle objectTypeHandle, Type objectType, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator xmlWriter, Object obj, Boolean isDeclaredType, Boolean writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle)
   at WriteArrayOfHistoryEventToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , CollectionDataContract )
   at System.Runtime.Serialization.CollectionDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator xmlWriter, Object obj, Boolean isDeclaredType, Boolean writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle)
   at WriteHistoryStateToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , ClassDataContract )
   at System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithXsiTypeAtTopLevel(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle originalDeclaredTypeHandle, Type graphType)
   at System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.DataContractSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObject(XmlDictionaryWriter writer, Object graph)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObject(Stream stream, Object graph)
   at DurableTask.Netherite.Serializer.SerializeTrackedObject(TrackedObject trackedObject) in /_/src/DurableTask.Netherite/Util/Serializer.cs:line 64
   at DurableTask.Netherite.Faster.FasterKV.Functions.FASTER.core.IFunctions<DurableTask.Netherite.Faster.FasterKV.Key,DurableTask.Netherite.Faster.FasterKV.Value,DurableTask.Netherite.EffectTracker,DurableTask.Netherite.Faster.FasterKV.Output,System.Object>.SingleReader(Key& key, EffectTracker& tracker, Value& src, Output& dst, ReadInfo& readInfo) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 1789
   at FASTER.core.ClientSession`6.InternalFasterSession.SingleReader(Key& key, Input& input, Value& value, Output& dst, RecordInfo& recordInfo, ReadInfo& readInfo)
   at DurableTask.Netherite.Faster.FasterKV.TryRead(PartitionReadEvent readEvent, EffectTracker effectTracker, Key key) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 708
   at DurableTask.Netherite.Faster.FasterKV.Read(PartitionReadEvent readEvent, EffectTracker effectTracker) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 694
   at DurableTask.Netherite.Faster.StoreWorker.Process(IList`1 batch) in /_/src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs:line 389 terminatePartition=True```

Any details available.

As Netherite backend is GA and much faster, and apt to scale out better, than default, trying to justify moving our banking project to it.

The text was updated successfully, but these errors were encountered:

sebastianburckhardt · 2023-04-28T22:20:17Z

I have been seeing this type of problem in my tests also. It's a System.OutOfMemoryException which is unfortunately not so easy to track down. I did find some memory leaks and am currently working on PRs to fix them. But it is not clear yet that I have found the real source of the problem.

davidmrdavid · 2023-04-28T22:27:25Z

Hi @ericleigh007 - thank you for reaching out.
I'm looking to reproduce this on my end. Can you tell me a bit more about how you're setting up this load test, or share any reproducers projects you may have handy? Thanks!

ericleigh007 · 2023-04-29T11:06:17Z

I published this on github here:
https://github.com/ericleigh007/DurableFunctionBenchmark-Ne/tree/master

I did upgrade to the latest Netherite extension before trying again, and I immediately got the Out of Memory exception.

I now wonder if things will be different if I follow any steps, such as redeploying my event hub. I've already used the included "Cleanup Function" in order to re-create the storage account, and that didn't have an effect.

Unlike some of the other problems I'd reported before with this same benchmark, this one appears to be a show-stopper. When it occurs, the entire benchmark fails, with no retries -- nothing, and for some reason, it is happening much more often.

Any workaround I should try, that you've discovered?

ericleigh007 · 2023-04-29T12:21:51Z

Latest crash, in context, showing the orchestrators beginning to launch, but encountering this issue.

2023-04-29T12:18:00Z   [Warning]   BandSectionSubOrchestrator starting orchestrator for RunId:348cc481-0e86-42c1-a417-4fa313ec4dcd, #4, launching 10 activities
2023-04-29T12:18:00Z   [Warning]   BandSectionSubOrchestrator starting orchestrator for RunId:348cc481-0e86-42c1-a417-4fa313ec4dcd, #19, launching 10 activities
2023-04-29T12:18:00Z   [Warning]   BandSectionSubOrchestrator starting orchestrator for RunId:348cc481-0e86-42c1-a417-4fa313ec4dcd, #2, launching 10 activities
2023-04-29T12:18:00Z   [Warning]   BandSectionSubOrchestrator starting orchestrator for RunId:348cc481-0e86-42c1-a417-4fa313ec4dcd, #5, launching 10 activities
2023-04-29T12:18:00Z   [Warning]   BandSectionSubOrchestrator starting orchestrator for RunId:348cc481-0e86-42c1-a417-4fa313ec4dcd, #14, launching 10 activities
2023-04-29T12:18:00Z   [Warning]   BandSectionSubOrchestrator starting orchestrator for RunId:348cc481-0e86-42c1-a417-4fa313ec4dcd, #9, launching 10 activities
2023-04-29T12:18:27Z   [Error]   Part03 !!! Encountered exception while working on store in StoreWorker.Process: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Xml.XmlStreamNodeWriter.FlushBuffer()
   at System.Xml.XmlStreamNodeWriter.UnsafeWriteUTF8Chars(Char* chars, Int32 charCount)
   at System.Xml.XmlUTF8NodeWriter.UnsafeWriteEscapedText(Char* chars, Int32 count)
   at System.Xml.XmlUTF8NodeWriter.WriteEscapedText(String s)
   at System.Xml.XmlBaseWriter.WriteString(String value)
   at System.Xml.XmlBaseWriter.WriteValue(String value)
   at System.Xml.XmlDictionaryAsyncCheckWriter.WriteValue(String value)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteString(XmlWriterDelegator xmlWriter, String value, XmlDictionaryString name, XmlDictionaryString ns)
   at WriteTaskCompletedEventToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , ClassDataContract )
   at System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithXsiType(XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle objectTypeHandle, Type objectType, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator xmlWriter, Object obj, Boolean isDeclaredType, Boolean writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle)
   at WriteArrayOfHistoryEventToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , CollectionDataContract )
   at System.Runtime.Serialization.CollectionDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator xmlWriter, Object obj, Boolean isDeclaredType, Boolean writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle)
   at WriteHistoryStateToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , ClassDataContract )
   at System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithXsiTypeAtTopLevel(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle originalDeclaredTypeHandle, Type graphType)
   at System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.DataContractSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObject(XmlDictionaryWriter writer, Object graph)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObject(Stream stream, Object graph)
   at DurableTask.Netherite.Serializer.SerializeTrackedObject(TrackedObject trackedObject) in /_/src/DurableTask.Netherite/Util/Serializer.cs:line 64
   at DurableTask.Netherite.Faster.FasterKV.Functions.FASTER.core.IFunctions<DurableTask.Netherite.Faster.FasterKV.Key,DurableTask.Netherite.Faster.FasterKV.Value,DurableTask.Netherite.EffectTracker,DurableTask.Netherite.Faster.FasterKV.Output,System.Object>.SingleReader(Key& key, EffectTracker& tracker, Value& src, Output& dst, ReadInfo& readInfo) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 1789
   at FASTER.core.ClientSession`6.InternalFasterSession.SingleReader(Key& key, Input& input, Value& value, Output& dst, RecordInfo& recordInfo, ReadInfo& readInfo)
   at DurableTask.Netherite.Faster.FasterKV.TryRead(PartitionReadEvent readEvent, EffectTracker effectTracker, Key key) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 708
   at DurableTask.Netherite.Faster.FasterKV.Read(PartitionReadEvent readEvent, EffectTracker effectTracker) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 694
   at DurableTask.Netherite.Faster.StoreWorker.Process(IList`1 batch) in /_/src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs:line 389 terminatePartition=True
2023-04-29T12:18:27Z   [Warning]   Part03 !!! EventHubsProcessor shut down before partition fully started in StartPartitionAsync:  terminatePartition=True```

ericleigh007 · 2023-04-29T17:27:05Z

After completely removing all of the taskhub containers from the storage account, I have not experienced this error yet.

davidmrdavid · 2023-05-01T23:56:05Z

Glad to hear you've found a workaround, @ericleigh007.
I'll be running your reproducer tomorrow and reporting back on any findings.

sebastianburckhardt · 2023-05-02T21:36:03Z

I am seeing OutOfMemory problems in some of my tests also but am having a very hard time pinning down the reason, so I can't really tell if it is the same reason or not.

davidmrdavid · 2023-05-03T00:51:45Z

@ericleigh007: I'm getting a 404 for this link: https://github.com/ericleigh007/DurableFunctionBenchmark-Ne/tree/master
Could it be that this is a private repo?

ericleigh007 · 2023-05-03T07:50:22Z

Big apologies there -- public now.

sebastianburckhardt · 2023-05-04T00:47:52Z

I found an issue that was introduced with the new Faster parameter settings introduced in 1.3.3 that caused an excessive amount of memory to be allocated and could be responsible for a lot of recent issues caused by OutOfMemory. This is an easy fix and I will try to release it ASAP since it appears to be affecting many of our users.

I also found several other issues that can cause memory problems which are a bit harder to fix and I will address those in additional work.

Also, just out of curiosity, are you purging or replacing old orchestration instances? Having the total number of orchestration or entity instances in storage continuously grow can create memory issues with Netherite also. You can use purge timer functions as described here: #229 (comment)
to periodically purge orchestration instances that have completed.

ericleigh007 · 2023-05-04T12:06:58Z

Hi Sir,
this seems like good news. Eager to try the new code when available.
As for purging -- yes that is something we do now with the default backend and would continue.
For my tests, i don't do any of that because i'm recreating the task hub often.

sebastianburckhardt · 2023-05-04T22:58:59Z

FYI, we just released 1.3.5 which has several fixes that could potentially help with this situation.

lukasportal · 2023-05-30T17:45:46Z

@sebastianburckhardt , we're using 1.3.5 and still seeing lots of OutOfMemory issues like this one

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.Xml.XmlDictionaryReader.ReadContentAsString(Int32 maxStringContentLength)
   at System.Xml.XmlBaseReader.ReadContentAsString()
   at System.Xml.XmlBaseReader.ReadElementContentAsString()
   at ReadExecutionStartedEventFromXml.ReadExecutionStartedEventFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
   at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
   at ReadArrayOfHistoryEventFromXml.ReadArrayOfHistoryEventFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString , XmlDictionaryString , CollectionDataContract )
   at System.Runtime.Serialization.CollectionDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
   at ReadHistoryStateFromXml.ReadHistoryStateFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
   at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
   at System.Runtime.Serialization.DataContractSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(XmlDictionaryReader reader)
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(Stream stream)
   at DurableTask.Netherite.Serializer.DeserializeTrackedObject(Byte[] bytes) in /_/src/DurableTask.Netherite/Util/Serializer.cs:line 70
   at DurableTask.Netherite.Faster.FasterKV.Functions.FASTER.core.IFunctions<DurableTask.Netherite.Faster.FasterKV.Key,DurableTask.Netherite.Faster.FasterKV.Value,DurableTask.Netherite.EffectTracker,DurableTask.Netherite.Faster.FasterKV.Output,System.Object>.CopyUpdater(Key& key, EffectTracker& tracker, Value& oldValue, Value& newValue, Output& output, RMWInfo& info) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 1743
   at FASTER.core.FasterKV`2.CreateNewRecordRMW[Input,Output,Context,FasterSession](Key& key, Input& input, Value& value, Output& output, PendingContext`3& pendingContext, FasterSession fasterSession, FasterExecutionContext`3 sessionCtx, HashBucket* bucket, Int32 slot, Int64 logicalAddress, Int64 physicalAddress, RecordInfo srcRecordInfo, UInt16 tag, HashBucketEntry entry, Int64 latestLogicalAddress, Int64 prevHighestReadCacheLogicalAddress, Int64 lowestReadCachePhysicalAddress, Int64 unsealLogicalAddress, Int64 unsealPhysicalAddress, Boolean doingCU, Boolean fromPending)
   at DurableTask.Netherite.Faster.FasterKV.PerformFasterRMWAsync(Key k, EffectTracker tracker) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 811
   at DurableTask.Netherite.Faster.FasterKV.ProcessEffectOnTrackedObject(Key k, EffectTracker tracker) in /_/src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs:line 791
   at DurableTask.Netherite.EffectTracker.<ProcessUpdate>g__ProcessRecursively|27_0() in /_/src/DurableTask.Netherite/Abstractions/PartitionState/EffectTracker.cs:line 141
   at DurableTask.Netherite.EffectTracker.<ProcessUpdate>g__ProcessRecursively|27_0() in /_/src/DurableTask.Netherite/Abstractions/PartitionState/EffectTracker.cs:line 146
   at DurableTask.Netherite.EffectTracker.ProcessUpdate(PartitionUpdateEvent updateEvent) in /_/src/DurableTask.Netherite/Abstractions/PartitionState/EffectTracker.cs:line 130

we're using basic plan, 1 unit and 12 partitions. This error happens to different partition numbers, not just this one. i've seen in 2, 5, 7, 9, etc

the azure functions app is on consumption linux plan

we're running tests with a relatively low amount of data being sent. about 10k messages with a few bytes each

sebastianburckhardt · 2023-07-08T14:35:52Z

The latest release 1.4.0 addresses this issue. It contains a blob-batching optimization #275 that improves the performance for cases where partitions or clients transmit medium to large amounts of data (in terms of either total size, or number of messages). It also adds better handling for Out-Of-Memory issues.

I think there is a good chance it will fix the issues you observed. If not, let me know and I will take another look.

microsoft-github-policy-service bot added the Needs: Triage 🔍 label Apr 28, 2023

davidmrdavid added bug Something isn't working and removed Needs: Triage 🔍 labels May 2, 2023

sebastianburckhardt mentioned this issue May 4, 2023

Fix excessive memory use introduced in release 1.3.3 #266

Merged

sebastianburckhardt added needs author response and removed bug Something isn't working labels Jul 8, 2023

ericleigh007 closed this as completed Jul 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash using medium test case #263

Crash using medium test case #263

ericleigh007 commented Apr 28, 2023

sebastianburckhardt commented Apr 28, 2023

davidmrdavid commented Apr 28, 2023

ericleigh007 commented Apr 29, 2023 •

edited

Loading

ericleigh007 commented Apr 29, 2023

ericleigh007 commented Apr 29, 2023

davidmrdavid commented May 1, 2023

sebastianburckhardt commented May 2, 2023

davidmrdavid commented May 3, 2023

ericleigh007 commented May 3, 2023

sebastianburckhardt commented May 4, 2023

ericleigh007 commented May 4, 2023

sebastianburckhardt commented May 4, 2023

lukasportal commented May 30, 2023

sebastianburckhardt commented Jul 8, 2023

Crash using medium test case #263

Crash using medium test case #263

Comments

ericleigh007 commented Apr 28, 2023

sebastianburckhardt commented Apr 28, 2023

davidmrdavid commented Apr 28, 2023

ericleigh007 commented Apr 29, 2023 • edited Loading

ericleigh007 commented Apr 29, 2023

ericleigh007 commented Apr 29, 2023

davidmrdavid commented May 1, 2023

sebastianburckhardt commented May 2, 2023

davidmrdavid commented May 3, 2023

ericleigh007 commented May 3, 2023

sebastianburckhardt commented May 4, 2023

ericleigh007 commented May 4, 2023

sebastianburckhardt commented May 4, 2023

lukasportal commented May 30, 2023

sebastianburckhardt commented Jul 8, 2023

ericleigh007 commented Apr 29, 2023 •

edited

Loading