[Backport 2.x] Add heuristic slice_size for slice level top buckets in concurrent segment search (#11585)

[jed326]

Description

Backport 074bc6a from #11585.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change bffc471

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git]

[github-actions]

❕ Gradle check result for bffc471: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testList

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2cbe873) 71.15% compared to head (bffc471) 70.97%.

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #11620      +/-   ##
============================================
- Coverage     71.15%   70.97%   -0.18%     
+ Complexity    59350    59231     -119     
============================================
  Files          4883     4883              
  Lines        279308   279308              
  Branches      40934    40934              
============================================
- Hits         198729   198233     -496     
- Misses        63738    64343     +605     
+ Partials      16841    16732     -109     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.