Single Partition Queries that resume using continuation token after partition split can observe failure or sometimes wrong results on SDK v3.38 and beyond

[adityasa]

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug
This issue affects a limited set of queries on SDK v3.38 and beyond:

• queries that target a single partition
• queries that don't explicitly target a single partition, but collection itself has a single physical partition
  Such queries are executed using ODE codepath starting SDK v3.38 and beyond.

If a continuation token is acquired from such queries and queries are resumed using this continuation token after a partition split has occurred (which means they are no longer eligible for ODE codepath), the query execution fails with MalformedContinuationToken exception.

To Reproduce
Execute a query targeting single partition so that continuation token is produced. Capture the continuation token. Pause query execution. Make changes to collection such that partition split occurs. Resume the query execution.

Expected behavior
Query execution succeeds and correct results are returned.

Actual behavior
Error of following form is produced:
Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: BadRequest (400); Substatus: 20007; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 20007; ActivityId: ; Reason: (Invalid format for continuation token {"OptimisticDirectExecutionToken":{"token":". . .
In some rare cases error is not observed but the results may be incorrect.

Environment summary
SDK Version: v3.38 and beyond
OS Version (e.g. Windows, Linux, MacOSX)

Additional context

Mitigation
Turn off ODE for query execution. This can be done

• either by setting EnableOptimisticDirectExecution to false
  https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/performance-tips-query-sdk?tabs=v3&pivots=programming-language-csharp#optimizing-single-partition-queries-with-optimistic-direct-execution
• or by setting environment variable AZURE_COSMOS_OPTIMISTIC_DIRECT_EXECUTION_ENABLED to false on machines executing the query using Cosmos DB SDK

[adityasa]

This is being actively investigated.

[adityasa]

This issue is fixed in v3.39.2