Clearing state on a grain which hasn't been saved to database yet causes AdoNetStorageProvider to crash

[Arshia001]

Calling ClearStateAsync on an AdoNetStorageProvider instance for a grain that has no database entry (no call to WriteStateAsync yet) causes an exception:

Exc level 0: Orleans.Storage.InconsistentStateException: Version conflict (ClearState): ServiceId=xxxxxxxx ProviderName=xxxxxxxx GrainType=xxxxxxxx GrainId=0 ETag=.
   at Orleans.Storage.AdoNetStorageProvider.<ClearStateAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Core.GrainStateStorageBridge.<ClearStateAsync>d__8.MoveNext()
Exc level 0: Orleans.Storage.InconsistentStateException: Version conflict (ClearState): ServiceId=xxxxxxxx ProviderName=xxxxxxxx GrainType=xxxxxxxx GrainId=0 ETag=.
   at Orleans.Storage.AdoNetStorageProvider.<ClearStateAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Core.GrainStateStorageBridge.<ClearStateAsync>d__8.MoveNext()

[sergeybykov]

Please let us know if this is still an issue in 2.1.

[Arshia001]

@sergeybykov This still happens in 2.1. Here's the exception output:

fail: Orleans.Storage.AdoNetGrainStorage[102204]
      Error from storage provider AdoNetGrainStorage.Grains.TestGrain during ClearState for grain Type=Grains.TestGrain Pk=*grn/F408F887/0000000000000000000000000000000103fffffff408f887-0xAFBADDC3 Id=GrainReference:*grn/F408F887/00000001 Error=

      Exc level 0: Orleans.Storage.InconsistentStateException: Version conict (ClearState): ServiceId=Service ProviderName=Default GrainType=Grains.TestGrain GrainId=1 ETag=.
         at Orleans.Storage.AdoNetGrainStorage.ClearStateAsync(String grainType, GrainReference grainReference, IGrainState grainState) in D:\build\agent\_work\17\s\src\AdoNet\Orleans.Persistence.AdoNet\Storage\Provider\AdoNetGrainStorage.cs:line 193
         at Orleans.Core.StateStorageBridge`1.ClearStateAsync() in D:\build\agent\_work\17\s\src\Orleans.Runtime\Storage\StateStorageBridge.cs:line 130
InconsistentStateException: Version conict (ClearState): ServiceId=Service ProviderName=Default GrainType=Grains.TestGrain GrainId=1 ETag=. Expected Etag= Received Etag=

[ReubenBond]

@veikkoeeva are you able to take a look at this?

[veikkoeeva]

@ReubenBond Likely around the first week in February. If I understand correctly, a test case to replicate this is to arrange so that state is cleared without first writing. Should be added to test cases too.

Peculiar if this hasn't affected many more people.

< Edit, maybe this week. Can't promise, though. :)

[veikkoeeva]

@ReubenBond Now that I thought about this (in sauna), if there is an attempt to perform any other operation than write when version is null, could that be just a no-op on some Orleans layer before persistence providers? Is there a case where it should behave in certain way in this situation and if so, can it be enforced before reaching a persistence provider?

Since the fix could be to check that in the provider or the latest in the database.

[cda963]

Any news on this? Still happening on 3.3.0

[veikkoeeva]

@cojocaru-dragos-alexandru I have missed this since writing this at least.

@ReubenBond Do you remember what other providers do in this situation and if it is undefined? I mean it is InconsistentStateException. :)

It looks like to me the fix is a no-op in the provider in case Orleans calls in the provider null as a version number (ETag) and return default as the state. This applies only to clearing the state but not to reading or writing the state.

If so, I can take a look at this next weekend (knocking on the wood I remember this time).

[OracPrime]

This also seems to be the case with AzureBlobStorage running against Azurite.

fail: Orleans.Storage.AzureBlobGrainStorage[200314]
Error clearing: GrainType=Namespace.ParameterDataSliceGrain1[[System.Int32]],Namespace.parameterdataslicegrain Grainid=GrainReference:*grn/Namespace.ParameterDataSliceGrain1/0+{"JobId":"378585c0-e079-4441-ba86-c2c8a1d510f0","ParameterId":"8a6ae96d-4fc7-5a75-87a0-6a179dd80fa8","DimensionLevels":[]}<[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]> ETag= BlobName=Namespace.ParameterDataSliceGrain1[[System.Int32]],Namespace.parameterdataslicegrain-GrainReference=000000000000000000000000000000000603c6e3511ad079+{"JobId":"378585c0-e079-4441-ba86-c2c8a1d510f0","ParameterId":"8a6ae96d-4fc7-5a75-87a0-6a179dd80fa8","DimensionLevels":[]} GenericArguments=[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e].json in Container=grains Exception=Blob storage condition not Satisfied. BlobName: Namespace.ParameterDataSliceGrain1[[System.Int32]],Namespace.parameterdataslicegrain-GrainReference=000000000000000000000000000000000603c6e3511ad079+{"JobId":"378585c0-e079-4441-ba86-c2c8a1d510f0","ParameterId":"8a6ae96d-4fc7-5a75-87a0-6a179dd80fa8","DimensionLevels":[]} GenericArguments=[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e].json, Container: grains, CurrentETag:
InconsistentStateException: Blob storage condition not Satisfied. BlobName: Namespace.ParameterDataSliceGrain`1[[System.Int32]],Namespace.parameterdataslicegrain-GrainReference=000000000000000000000000000000000603c6e3511ad079+{"JobId":"378585c0-e079-4441-ba86-c2c8a1d510f0","ParameterId":"8a6ae96d-4fc7-5a75-87a0-6a179dd80fa8","DimensionLevels":[]} GenericArguments=[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e].json, Container: grains, CurrentETag: Expected Etag=Unknown Received Etag= Azure.RequestFailedException: The condition specified using HTTP conditional header(s) is not met.
RequestId:9ed728b2-f43b-4c99-ba4e-e71574a1d964
Time:2021-01-16T13:21:50.455Z
Status: 412 (The condition specified using HTTP conditional header(s) is not met.)
ErrorCode: ConditionNotMet

  Headers:
  Server: Azurite-Blob/3.10.0
  x-ms-error-code: ConditionNotMet
  x-ms-request-id: 9ed728b2-f43b-4c99-ba4e-e71574a1d964
  Date: Sat, 16 Jan 2021 13:21:50 GMT
  Connection: keep-alive
  Transfer-Encoding: chunked
  Content-Type: application/xml

     at Azure.Storage.Blobs.BlobRestClient.Blob.DeleteAsync_CreateResponse(ClientDiagnostics clientDiagnostics, Response response)
     at Azure.Storage.Blobs.BlobRestClient.Blob.DeleteAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, String version, String snapshot, String versionId, Nullable`1 timeout, String leaseId, Nullable`1 deleteSnapshots, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, Nullable`1 ifMatch, Nullable`1 ifNoneMatch, String ifTags, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)
     at Azure.Storage.Blobs.Specialized.BlobBaseClient.DeleteInternal(DeleteSnapshotsOption snapshotsOption, BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken, String operationName)
     at Azure.Storage.Blobs.Specialized.BlobBaseClient.DeleteIfExistsInternal(DeleteSnapshotsOption snapshotsOption, BlobRequestConditions conditions, Boolean async, CancellationToken cancellationToken)
     at Azure.Storage.Blobs.Specialized.BlobBaseClient.DeleteIfExistsAsync(DeleteSnapshotsOption snapshotsOption, BlobRequestConditions conditions, CancellationToken cancellationToken)
     at Orleans.Storage.AzureBlobGrainStorage.DoOptimisticUpdate[TResult](Func`1 updateOperation, BlobClient blob, String currentETag)