Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.ArgumentException in CosmosJsonDotNetSerializer #3369

Closed
ViconiaDeVier opened this issue Jul 29, 2022 · 5 comments
Closed

System.ArgumentException in CosmosJsonDotNetSerializer #3369

ViconiaDeVier opened this issue Jul 29, 2022 · 5 comments

Comments

@ViconiaDeVier
Copy link

ViconiaDeVier commented Jul 29, 2022

Describe the bug
After upgrading Microsoft.Azure.Cosmos library from 3.23.0-preview to 3.28.0-preview and Microsoft.Azure.Cosmos.Encryption.Custom library from 1.0.0-preview02 to 1.0.0-preview03 we started to see a lot of System.ArgumentException on our PROD env:
unhandled exception System.ArgumentException: Stream was not readable.
at System.IO.StreamReader..ctor(Stream stream, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean leaveOpen)
at Microsoft.Azure.Cosmos.Encryption.Custom.CosmosJsonDotNetSerializer.FromStream(Stream stream)
at Microsoft.Azure.Cosmos.Encryption.Custom.EncryptionFeedIterator.ConvertResponseToDecryptableItems(Stream content)
at await Microsoft.Azure.Cosmos.Encryption.Custom.EncryptionFeedIterator.ReadNextWithoutDecryptionAsync(?)
at await Microsoft.Azure.Cosmos.Encryption.Custom.EncryptionFeedIterator`1.ReadNextAsync(?)

I see that there was something similar fixed in this PR so I was wondering if it's possible that there can be similar issue related to CosmosJsonDotNetSerializer ?
It doesn't seem that it is happening because of some problem with particular items since they were read and decrypted successfully during another attempts.
So it looks that sometimes for some reason the stream is closed before the item was decrypted

Actual behavior
Stream is closed before item is decrypted

Environment summary
SDK Version: 3.28.0-preview
OS Version (e.g. Windows, Linux, MacOSX) Windows


Upd:
Some additional details which might help investigation.
This issue normally appears in East Japan. At one moment of time we see this error happens on different instances, processing one particular partition at one moment of time (but it's different partitions over the time). Also - that happens only for the cases when we connect to East Japan account as a read replica account.

@kr-santosh
Copy link
Contributor

ResponseMessage responseMessage = await this.feedIterator.ReadNextAsync(cancellationToken);
seems to return a closed stream. Need the Cosmos Core team to have a look at this. @sourabh1007

@kr-santosh
Copy link
Contributor

Hi @sboshra can the query team have a look at this issue.

@kirankumarkolli
Copy link
Member

Is the account strong/bounded-staleness?

@ViconiaDeVier
Copy link
Author

@kirankumarkolli the account is bounded-staleness

@kirankumarkolli
Copy link
Member

Related fix is released in 3.29-preview (discussed offline)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

6 participants