DurableTask.AzureStorage: Alternate fix for the 4 MB max entity size which covers more scenarios. #194
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cgillum
changed the title
kashimiz
reviewed
Jun 15, 2018
| // Assume at least 1 KB of data per entity to account for static-length properties | ||
| int estimatedByteCount = 1024; | ||
|
|
||
| // Count the bytes for variable-length properties, which are assumed to always be strings |
There was a problem hiding this comment.
I take it this is a safe assumption to make, but I'm curious as to why. Do all property values come in as serialized strings?
There was a problem hiding this comment.
Not all, but the ones in the VariableSizeEntityProperties collection will always be strings. With the exception of Name (and sometimes Result), these are all serialized JSON payloads. If we wanted, we could store these as binary data as well, but it's much easier to debug things when they are strings.
| EntityProperty property; | ||
| if (entity.Properties.TryGetValue(propertyName, out property) && !string.IsNullOrEmpty(property.StringValue)) | ||
| { | ||
| estimatedByteCount += Encoding.Unicode.GetByteCount(property.StringValue); |
There was a problem hiding this comment.
Curious; why UTF-16? Is this the format Table Storage uses?
There was a problem hiding this comment.
Yep. This basically means 2 bytes for every character in a string.
kashimiz
approved these changes
Jun 15, 2018
simonporter
pushed a commit
that referenced
this pull request
Jun 19, 2018
* DurableTask.AzureStorage API to enumerate instances (#187) * DurableTask.AzureStorage ETW trace improvements (#192) * Adding 4MB limit check for Azure Storage Table batch insert (#191) * DurableTask.AzureStorage: Alternate fix for the 4 MB max entity size which covers more scenarios. (#194) * Updated Newtonsoft.Json to v11.0.2, WindowsAzure.Storage to v8.6.0. (#193) * Fixed issues with the ETW event source and added test reliability improvements.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a revised version of a previous pull request: #191. After merging, I did some testing and found a few issues. This new version addresses those issues.
FYI @tohling, @gled4er
This addresses Azure/azure-functions-durable-extension#339.