Error Occured in Bulk OperationError: Partition key error. Either the partitions have split or an operation has an unsupported partitionKey type

[developer-ankursingh]

Azure based dependencies being used -
@azure/cosmos":"^3.15.1"
@azure/identity": "^2.0.1"

Describe the bug
We have been using Azure Cosmos library with Azure identity liberary for RBAC mechanism to perform bulk delete operation. Till 2 days back , it was working fine so we were able to delete multiple records from Cosmos DB. But now we are getting a weird error/exception each time we try to delete even a single record. As of now, the no. of records is around 2.3 millions in the container in which we are trying to perform deletion. We have validated multiple times that we are passing correct data to bulk delete API (uniqueId, PartitionKey & Operation Type - Delete ) even then we are facing above issue . Same Bulk delete API is working fine for other containers having much lesser amount of data .

To Reproduce
Steps to reproduce the behavior:
1.

Expected behavior
We should be able to perform deletion sucessfully no matter how many records we have in the container.

Screenshots
Added screenshot to help explain our problem.

    `   while (i) {
           var pos = data.length - i;
          var superItem = data[pos];
        operations.push({
            operationType: "Delete",
            id: superItem.id,
            partitionKey: superItem.partitionKey
        })

        if (i % 100 === 0 || i === 1) {
            try {
                var response = await createdContainer.items.bulk(operations, { continueOnError: true });
                throttledCount = response.filter((r) => r.statusCode === 429).length;
                while (throttledCount !== 0) {
                    response = await createdContainer.items.bulk(operations, { continueOnError: true });
                    throttledCount = response.filter((r) => r.statusCode === 429).length;
                }
            } catch (error) {
                console.log("Error Occured in Bulk Operation" + error);
            }

            operations = [];
            if (i === 0) {
                console.log('100 records got deleted')
            }
        }
        i--;
    }`

Additional context
As part of a validation check, we tried to delete records for other containers in our DEV & SIT env and the records got deleted successfully. We did an analysis for the above issue and couldn’t find much on internet also except the below links.

#18682

azure-sdk-for-js/sdk/cosmosdb/cosmos/src/client/Item/Items.ts

Line 463 in 3cfbb8b

"Partition key error. Either the partitions have split or an operation has an unsupported partitionKey type"

Using above links, we can see the code written in Azure Cosmos Node JS library and the exception is being thrown from the library itself based on some internal min-max range for partition key. From our side, we cannot do much as of now to debug this further so raising a ticket here for further help.

[developer-ankursingh]

FYI, we are able to perform bulk update operation successfully but the bulk delete operation is throwing error for the same container.
I have attached debug logs & screenshots for both Update & Delete operations for your reference.

[developer-ankursingh]

attached logs as wel
bulk_delete_error_logs.txt
l

[kushagraThapar]

@developer-ankursingh - we are investigating this issue. @sajeetharan can you please look into this ?

[developer-ankursingh]

Thanks for the update @sajeetharan . We can also have a session if possible with your team to demo the issue. Please let us know since it is impacting our project developement .

[sajeetharan]

@developer-ankursingh From the image looks like this happens in the stage environment right? We have checked the backend logs, we are able to see the 410 errors as well on this account on 02/02/22, just wanted to confirm if this is only for deletion. We'll schedule a call to understand once we analyze the code and check with backend team.

[developer-ankursingh]

@sajeetharan It is happening in SIT & STAGING but not in DEV. Just to confirm this isssue pops up only for DELETE operation for SIT & STAGING containers only but same code is working fine for other containers (all environments). It would be really nice if we can have a call where we can give you a walkthrough of the code.

[sajeetharan]

@developer-ankursingh Sorry for the delayed response, we were able to identify your requests in the backend and it looks like an issue to be on the SDK, which related to cache which maintains the partition key range. We will work on this fix soon.

[developer-ankursingh]

@sajeetharan Thank you for the update ! Let us know whenever you push out the fix so that we can validate and confirm.

[sajeetharan]

Suggestions to resolve the issue,

1. The SDK reads the entire partition key map on every bulk operation. It's never cached which is a huge issue for large collections with many partitions. Loading the ranges can easily become much more expensive than making the bulk operation itself. The fix is to cache the ranges on the container object so they are only loaded once.

2. The SDK explicitly throws an error on bulk operations during a partition split. It needs to handle the 410 gracefully by retrying the failed operations AFTER invalidating the cache partition map. But first you have to fix (1) above.

An even better option would be to fix this in the gateway. When the backend bubbles up a 410 to the gateway during bulk it could refresh its own internal partition map and reroute to the correct partition.

@jay-most @JericHunter

[jay-most]

Thanks @sajeetharan! @developer-ankursingh reopen if needed.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @bkolant-MSFT, @sajeetharan, @pjohari-ms.

Issue Details

---

Azure based dependencies being used -
@azure/cosmos":"^3.15.1"
@azure/identity": "^2.0.1"

Describe the bug
We have been using Azure Cosmos library with Azure identity liberary for RBAC mechanism to perform bulk delete operation. Till 2 days back , it was working fine so we were able to delete multiple records from Cosmos DB. But now we are getting a weird error/exception each time we try to delete even a single record. As of now, the no. of records is around 2.3 millions in the container in which we are trying to perform deletion. We have validated multiple times that we are passing correct data to bulk delete API (uniqueId, PartitionKey & Operation Type - Delete ) even then we are facing above issue . Same Bulk delete API is working fine for other containers having much lesser amount of data .

To Reproduce
Steps to reproduce the behavior:
1.

Expected behavior
We should be able to perform deletion sucessfully no matter how many records we have in the container.

Screenshots
Added screenshot to help explain our problem.

    `   while (i) {
           var pos = data.length - i;
          var superItem = data[pos];
        operations.push({
            operationType: "Delete",
            id: superItem.id,
            partitionKey: superItem.partitionKey
        })

        if (i % 100 === 0 || i === 1) {
            try {
                var response = await createdContainer.items.bulk(operations, { continueOnError: true });
                throttledCount = response.filter((r) => r.statusCode === 429).length;
                while (throttledCount !== 0) {
                    response = await createdContainer.items.bulk(operations, { continueOnError: true });
                    throttledCount = response.filter((r) => r.statusCode === 429).length;
                }
            } catch (error) {
                console.log("Error Occured in Bulk Operation" + error);
            }

            operations = [];
            if (i === 0) {
                console.log('100 records got deleted')
            }
        }
        i--;
    }`

Additional context
As part of a validation check, we tried to delete records for other containers in our DEV & SIT env and the records got deleted successfully. We did an analysis for the above issue and couldn’t find much on internet also except the below links.

#18682

azure-sdk-for-js/sdk/cosmosdb/cosmos/src/client/Item/Items.ts

Line 463 in 3cfbb8b

"Partition key error. Either the partitions have split or an operation has an unsupported partitionKey type"

Using above links, we can see the code written in Azure Cosmos Node JS library and the exception is being thrown from the library itself based on some internal min-max range for partition key. From our side, we cannot do much as of now to debug this further so raising a ticket here for further help.

Author:	developer-ankursingh
Assignees:	jay-most
Labels:	bug, customer-reported, Client, Cosmos, Service Attention
Milestone:	-

[github-actions]

Hi @developer-ankursingh, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.