[BUG] Using an excessively large reindex slice can lead to a JVM OutOfMemoryError on coordinator

[kkewwei]

Describe the bug

If user set the slices to an excessively large value, it may result in a JVM OutOfMemoryError (OOM) on the coordinator.

POST _reindex?slices=10000
{
   "source":{
      "index":"index1"
   },
   "dest":{
      "index":"index2"
   }
}

In coordinator, the reindex on coordinator will be create 10000 slice query to the cluster.

OpenSearch/modules/reindex/src/main/java/org/opensearch/index/reindex/BulkByScrollParallelizationHelper.java

Line 196 in 696ed65

for (final SearchRequest slice : sliceIntoSubRequests(request.getSearchRequest(), IdFieldMapper.NAME, totalSlices)) {

.

As the index.max_slices_per_scroll is 1024, it will throw the exception:

ShardSearchFailure(lllegalArgumentException[The number of slices [10000] is too large. It must be less than [1024])

As the check occurs on every shard, if the shard number of the index1 is 100, it will create 1,000,000(10000*100) ShardSearchFailure.

In our production, the heap size of ShardSearchFailure is 26gb

Related component

Other

To Reproduce

1. create index:

PUT logs1
{
    "settings": {
        "index.number_of_shards": 100
    }
}

2. reindex

POST _reindex?slices=10000
{
   "source":{
      "index":"logs1"
   },
   "dest":{
      "index":"logs2"
   }
}

It will create 1m(100*10000) ShardSearchFailures

Expected behavior

We should check the slices on the coordinator, not on the every shard.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[sunqijun1]

@kkewwei Can this issue be assigned to me? I am very interested in it.