Merge branch 'main' into aggregation-precomputation-rare-terms

Signed-off-by: Anthony Leong <aj.leong623@gmail.com>

Author: ajleong623

.github/workflows/stalled.yml

@@ -17,7 +17,7 @@ jobs:
           private_key: ${{ secrets.APP_PRIVATE_KEY }}
           installation_id: 22958780
       - name: Stale PRs
-        uses: actions/stale@v9
+        uses: actions/stale@v10
         with:
           repo-token: ${{ steps.github_app_token.outputs.token }}
           stale-pr-label: 'stalled'

CHANGELOG.md

@@ -8,25 +8,38 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Expand fetch phase profiling to support inner hits and top hits aggregation phases  ([##18936](https://github.com/opensearch-project/OpenSearch/pull/18936))
 - Add temporal routing processors for time-based document routing ([#18920](https://github.com/opensearch-project/OpenSearch/issues/18920))
 - Implement Query Rewriting Infrastructure ([#19060](https://github.com/opensearch-project/OpenSearch/pull/19060))
-- The dynamic mapping parameter supports false_allow_templates ([#19065](https://github.com/opensearch-project/OpenSearch/pull/19065))
+- The dynamic mapping parameter supports false_allow_templates ([#19065](https://github.com/opensearch-project/OpenSearch/pull/19065) ([#19097](https://github.com/opensearch-project/OpenSearch/pull/19097)))
 - Add a toBuilder method in EngineConfig to support easy modification of configs([#19054](https://github.com/opensearch-project/OpenSearch/pull/19054))
 - Add StoreFactory plugin interface for custom Store implementations([#19091](https://github.com/opensearch-project/OpenSearch/pull/19091))
 - Use S3CrtClient for higher throughput while uploading files to S3 ([#18800](https://github.com/opensearch-project/OpenSearch/pull/18800))
 - Add a dynamic setting to change skip_cache_factor and min_frequency for querycache ([#18351](https://github.com/opensearch-project/OpenSearch/issues/18351))
 - Add overload constructor for Translog to accept Channel Factory as a parameter ([#18918](https://github.com/opensearch-project/OpenSearch/pull/18918))
 - Add subdirectory-aware store module with recovery support ([#19132](https://github.com/opensearch-project/OpenSearch/pull/19132))
+- Field collapsing supports search_after ([#19261](https://github.com/opensearch-project/OpenSearch/pull/19261))
 - Add a dynamic cluster setting to control the enablement of the merged segment warmer ([#18929](https://github.com/opensearch-project/OpenSearch/pull/18929))
+- Publish transport-grpc-spi exposing QueryBuilderProtoConverter and QueryBuilderProtoConverterRegistry ([#18949](https://github.com/opensearch-project/OpenSearch/pull/18949))
 - Support system generated search pipeline. ([#19128](https://github.com/opensearch-project/OpenSearch/pull/19128))
+- Add `epoch_micros` date format ([#14669](https://github.com/opensearch-project/OpenSearch/issues/14669))
+- Grok processor supports capturing multiple values for same field name ([#18799](https://github.com/opensearch-project/OpenSearch/pull/18799))
+- Upgrade opensearch-protobufs dependency to 0.13.0 and update transport-grpc module compatibility ([#19007](https://github.com/opensearch-project/OpenSearch/issues/19007))
+- Add new extensible method to DocRequest to specify type ([#19313](https://github.com/opensearch-project/OpenSearch/pull/19313))
+- [Rule based auto-tagging] Add Rule based auto-tagging IT ([#18550](https://github.com/opensearch-project/OpenSearch/pull/18550))
+- Add all-active ingestion as docrep equivalent in pull-based ingestion ([#19316](https://github.com/opensearch-project/OpenSearch/pull/19316))
 - Added precomputation for rare terms aggregation ([##18978](https://github.com/opensearch-project/OpenSearch/pull/18978))
 
 ### Changed
+- Refactor `if-else` chains to use `Java 17 pattern matching switch expressions`(([#18965](https://github.com/opensearch-project/OpenSearch/pull/18965))
 - Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl ([#18998](https://github.com/opensearch-project/OpenSearch/pull/18998))
 - IllegalArgumentException when scroll ID references a node not found in Cluster ([#19031](https://github.com/opensearch-project/OpenSearch/pull/19031))
 - Adding ScriptedAvg class to painless spi to allowlist usage from plugins ([#19006](https://github.com/opensearch-project/OpenSearch/pull/19006))
 - Replace centos:8 with almalinux:8 since centos docker images are deprecated ([#19154](https://github.com/opensearch-project/OpenSearch/pull/19154))
 - Add CompletionStage variants to IndicesAdminClient as an alternative to ActionListener ([#19161](https://github.com/opensearch-project/OpenSearch/pull/19161))
 - Remove cap on Java version used by forbidden APIs ([#19163](https://github.com/opensearch-project/OpenSearch/pull/19163))
 - Disable pruning for `doc_values` for the wildcard field mapper ([#18568](https://github.com/opensearch-project/OpenSearch/pull/18568))
+- Make all methods in Engine.Result public ([#19276](https://github.com/opensearch-project/OpenSearch/pull/19275))
+- Create and attach interclusterTest and yamlRestTest code coverage reports to gradle check task([#19165](https://github.com/opensearch-project/OpenSearch/pull/19165))
+- Optimized date histogram aggregations by preventing unnecessary object allocations in date rounding utils ([19088](https://github.com/opensearch-project/OpenSearch/pull/19088))
+- Optimize source conversion in gRPC search hits using zero-copy BytesRef ([#19280](https://github.com/opensearch-project/OpenSearch/pull/19280))
 
 ### Fixed
 - Fix unnecessary refreshes on update preparation failures ([#15261](https://github.com/opensearch-project/OpenSearch/issues/15261))
@@ -43,17 +56,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix QueryPhaseResultConsumer incomplete callback loops ([#19231](https://github.com/opensearch-project/OpenSearch/pull/19231))
 - Fix the `scaled_float` precision issue ([#19188](https://github.com/opensearch-project/OpenSearch/pull/19188))
 - Fix Using an excessively large reindex slice can lead to a JVM OutOfMemoryError on coordinator.([#18964](https://github.com/opensearch-project/OpenSearch/pull/18964))
+- [Flaky Test] Fix flaky test in SecureReactorNetty4HttpServerTransportTests with reproducible seed ([#19327](https://github.com/opensearch-project/OpenSearch/pull/19327))
+- Remove unnecessary looping in field data cache clear ([#19116](https://github.com/opensearch-project/OpenSearch/pull/19116))
+
 
 ### Dependencies
 - Bump `com.netflix.nebula.ospackage-base` from 12.0.0 to 12.1.0 ([#19019](https://github.com/opensearch-project/OpenSearch/pull/19019))
 - Bump `actions/checkout` from 4 to 5 ([#19023](https://github.com/opensearch-project/OpenSearch/pull/19023))
 - Bump `commons-cli:commons-cli` from 1.9.0 to 1.10.0 ([#19021](https://github.com/opensearch-project/OpenSearch/pull/19021))
 - Bump `org.jline:jline` from 3.30.4 to 3.30.5 ([#19013](https://github.com/opensearch-project/OpenSearch/pull/19013))
-- Bump `com.github.spotbugs:spotbugs-annotations` from 4.9.3 to 4.9.4 ([#19015](https://github.com/opensearch-project/OpenSearch/pull/19015))
+- Bump `com.github.spotbugs:spotbugs-annotations` from 4.9.3 to 4.9.5 ([#19015](https://github.com/opensearch-project/OpenSearch/pull/19015), [#19294](https://github.com/opensearch-project/OpenSearch/pull/19294))
 - Bump `com.azure:azure-storage-common` from 12.29.1 to 12.30.2 ([#19016](https://github.com/opensearch-project/OpenSearch/pull/19016), [#19145](https://github.com/opensearch-project/OpenSearch/pull/19145))
 - Update OpenTelemetry to 1.53.0 and OpenTelemetry SemConv to 1.34.0 ([#19068](https://github.com/opensearch-project/OpenSearch/pull/19068))
 - Bump `1password/load-secrets-action` from 2 to 3 ([#19100](https://github.com/opensearch-project/OpenSearch/pull/19100))
-- Bump `com.nimbusds:nimbus-jose-jwt` from 10.3 to 10.4.2 ([#19099](https://github.com/opensearch-project/OpenSearch/pull/19099), [#19101](https://github.com/opensearch-project/OpenSearch/pull/19101))
+- Bump `com.nimbusds:nimbus-jose-jwt` from 10.3 to 10.5 ([#19099](https://github.com/opensearch-project/OpenSearch/pull/19099), [#19101](https://github.com/opensearch-project/OpenSearch/pull/19101), [#19254](https://github.com/opensearch-project/OpenSearch/pull/19254))
 - Bump netty from 4.1.121.Final to 4.1.125.Final ([#19103](https://github.com/opensearch-project/OpenSearch/pull/19103)) ([#19269](https://github.com/opensearch-project/OpenSearch/pull/19269)
 - Bump Google Cloud Storage SDK from 1.113.1 to 2.55.0 ([#18922](https://github.com/opensearch-project/OpenSearch/pull/18922))
 - Bump `com.google.auth:google-auth-library-oauth2-http` from 1.37.1 to 1.38.0 ([#19144](https://github.com/opensearch-project/OpenSearch/pull/19144))
@@ -74,6 +90,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.maxmind.geoip2:geoip2` from 4.3.1 to 4.4.0 ([#19205](https://github.com/opensearch-project/OpenSearch/pull/19205))
 - Bump `org.jboss.xnio:xnio-nio` from 3.8.16.Final to 3.8.17.Final ([#19252](https://github.com/opensearch-project/OpenSearch/pull/19252))
 - Bump `actions/setup-java` from 4 to 5 ([#19143](https://github.com/opensearch-project/OpenSearch/pull/19143))
+- Bump `com.google.code.gson:gson` from 2.13.1 to 2.13.2 ([#19290](https://github.com/opensearch-project/OpenSearch/pull/19290)) ([#19293](https://github.com/opensearch-project/OpenSearch/pull/19293))
+- Bump `actions/stale` from 9 to 10 ([#19292](https://github.com/opensearch-project/OpenSearch/pull/19292))
+- Bump `com.nimbusds:oauth2-oidc-sdk` from 11.25 to 11.28 ([#19291](https://github.com/opensearch-project/OpenSearch/pull/19291))
+- Bump `log4j2` from 2.21.0 to 2.25.1 ([#19184](https://github.com/opensearch-project/OpenSearch/pull/19184))
+- Bump Apache Lucene from 10.2.2 to 10.3.0 ([#19296](https://github.com/opensearch-project/OpenSearch/pull/19296))
+- Add com.google.code.gson:gson to the gradle version catalog ([#19328](https://github.com/opensearch-project/OpenSearch/pull/19328))
 
 ### Deprecated
 

README.md

@@ -2,10 +2,12 @@
   <img src="https://opensearch.org/assets/img/opensearch-logo-themed.svg" height="64px">
 </a>
 
+[![License](https://img.shields.io/badge/license-Apache%20v2-blue.svg)](https://github.com/opensearch-project/OpenSearch/blob/main/LICENSE.txt)
 [![LFX Health Score](https://insights.production.lfx.dev/api/badge/health-score?project=opensearch-foundation)](https://insights.linuxfoundation.org/project/opensearch-foundation)
 [![LFX Active Contributors](https://insights.production.lfx.dev/api/badge/active-contributors?project=opensearch-foundation&repos=https://github.com/opensearch-project/OpenSearch)](https://insights.linuxfoundation.org/project/opensearch-foundation/repository/opensearch-project-opensearch)
 [![Code Coverage](https://codecov.io/gh/opensearch-project/OpenSearch/branch/main/graph/badge.svg)](https://codecov.io/gh/opensearch-project/OpenSearch)
 ![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/opensearch-project/OpenSearch?sort=semver)
+[![Linkedin](https://img.shields.io/badge/Follow-Linkedin-blue)](https://www.linkedin.com/company/opensearch-project)
 
 - [Welcome!](#welcome)
 - [Project Resources](#project-resources)

gradle/code-coverage.gradle

@@ -52,14 +52,19 @@ allprojects {
         executionDataFiles.add("$buildDir/jacoco/javaRestTest.exec")
         sourceSetsList.add(sourceSets.javaRestTest)
       }
+      if (tasks.findByName('yamlRestTest')) {
+        executionDataFiles.add("$buildDir/jacoco/yamlRestTest.exec")
+        sourceSetsList.add(sourceSets.yamlRestTest)
+      }
       if (!executionDataFiles.isEmpty()) {
         executionData.setFrom(files(executionDataFiles).filter { it.exists() })
         sourceSets(*sourceSetsList)
       }
       onlyIf {
         file("$buildDir/jacoco/test.exec").exists() ||
           file("$buildDir/jacoco/internalClusterTest.exec").exists() ||
-          file("$buildDir/jacoco/javaRestTest.exec").exists()
+          file("$buildDir/jacoco/javaRestTest.exec").exists() ||
+          file("$buildDir/jacoco/yamlRestTest.exec").exists()
       }
     }
   }
@@ -71,17 +76,25 @@ if (System.getProperty("tests.coverage")) {
       testCodeCoverageReport(JacocoCoverageReport) {
         testSuiteName = "test"
       }
+      testCodeCoverageReportInternalClusterTest(JacocoCoverageReport) {
+        testSuiteName = "internalClusterTest"
+      }
       testCodeCoverageReportJavaRestTest(JacocoCoverageReport) {
         testSuiteName = "javaRestTest"
       }
+      testCodeCoverageReportYamlRestTest(JacocoCoverageReport) {
+        testSuiteName = "yamlRestTest"
+      }
     }
   }
 
   // Attach code coverage report task to Gradle check task
   project.getTasks().named(JavaBasePlugin.CHECK_TASK_NAME).configure {
     dependsOn(
       tasks.named('testCodeCoverageReport', JacocoReport),
-      tasks.named('testCodeCoverageReportJavaRestTest', JacocoReport)
+      tasks.named('testCodeCoverageReportInternalClusterTest', JacocoReport),
+      tasks.named('testCodeCoverageReportJavaRestTest', JacocoReport),
+      tasks.named('testCodeCoverageReportYamlRestTest', JacocoReport)
     )
   }
 }

gradle/libs.versions.toml

@@ -1,6 +1,6 @@
 [versions]
 opensearch        = "3.3.0"
-lucene            = "10.2.2"
+lucene            = "10.3.0"
 
 bundled_jdk_vendor = "adoptium"
 bundled_jdk = "24.0.2+12"
@@ -13,14 +13,18 @@ jackson_databind  = "2.18.2"
 snakeyaml         = "2.1"
 icu4j             = "77.1"
 supercsv          = "2.4.0"
-log4j             = "2.21.0"
+log4j             = "2.25.1"
+error_prone_annotations = "2.41.0"
+spotbugs_annotations = "4.9.4"
 slf4j             = "2.0.17"
 asm               = "9.7"
 jettison          = "1.5.4"
 woodstox          = "6.4.0"
 kotlin            = "1.7.10"
 antlr4            = "4.13.1"
 guava             = "33.2.1-jre"
+gson              = "2.13.2"
+opensearchprotobufs = "0.13.0"
 protobuf          = "3.25.8"
 jakarta_annotation = "1.3.5"
 google_http_client = "1.44.1"
@@ -106,6 +110,7 @@ bouncycastle-tls = { group = "org.bouncycastle", name = "bctls-fips", version.re
 bouncycastle-pkix = { group = "org.bouncycastle", name = "bcpkix-fips", version.ref = "bouncycastle_pkix"}
 bouncycastle-pg = { group = "org.bouncycastle", name = "bcpg-fips", version.ref = "bouncycastle_pg"}
 bouncycastle-util = { group = "org.bouncycastle", name = "bcutil-fips", version.ref = "bouncycastle_util"}
+gson = { group = "com.google.code.gson", name = "gson", version.ref = "gson" }
 hamcrest = { group = "org.hamcrest", name = "hamcrest", version.ref = "hamcrest" }
 hdrhistogram = { group = "org.hdrhistogram", name = "HdrHistogram", version.ref = "hdrhistogram" }
 jackson-annotation = { group = "com.fasterxml.jackson.core", name = "jackson-annotations", version.ref = "jackson" }

libs/common/src/main/java/org/opensearch/common/Numbers.java

@@ -83,44 +83,60 @@ public static boolean isValidDouble(double value) {
      *  stored value cannot be converted to a long that stores the exact same
      *  value. */
     public static long toLongExact(Number n) {
-        if (n instanceof Byte || n instanceof Short || n instanceof Integer || n instanceof Long) {
-            return n.longValue();
-        } else if (n instanceof Float || n instanceof Double) {
-            double d = n.doubleValue();
-            if (d != Math.round(d)) {
-                throw new IllegalArgumentException(n + " is not an integer value");
+        return switch (n) {
+            case Byte b -> b.longValue();
+            case Short s -> s.longValue();
+            case Integer i -> i.longValue();
+            case Long l -> l;
+            case Float f -> {
+                double d = f.doubleValue();
+                if (d != Math.round(d)) {
+                    throw new IllegalArgumentException(f + " is not an integer value");
+                }
+                yield f.longValue();
             }
-            return n.longValue();
-        } else if (n instanceof BigDecimal) {
-            return ((BigDecimal) n).toBigIntegerExact().longValueExact();
-        } else if (n instanceof BigInteger) {
-            return ((BigInteger) n).longValueExact();
-        } else {
-            throw new IllegalArgumentException(
+            case Double d -> {
+                if (d != Math.round(d)) {
+                    throw new IllegalArgumentException(d + " is not an integer value");
+                }
+                yield d.longValue();
+            }
+            case BigDecimal bd -> bd.toBigIntegerExact().longValueExact();
+            case BigInteger bi -> bi.longValueExact();
+            default -> throw new IllegalArgumentException(
                 "Cannot check whether [" + n + "] of class [" + n.getClass().getName() + "] is actually a long"
             );
-        }
+        };
     }
 
     /** Return the {@link BigInteger} that {@code n} stores, or throws an exception if the
      *  stored value cannot be converted to a {@link BigInteger} that stores the exact same
      *  value. */
     public static BigInteger toBigIntegerExact(Number n) {
-        if (n instanceof Byte || n instanceof Short || n instanceof Integer || n instanceof Long) {
-            return BigInteger.valueOf(n.longValue());
-        } else if (n instanceof Float || n instanceof Double) {
-            double d = n.doubleValue();
-            if (d != Math.round(d)) {
-                throw new IllegalArgumentException(n + " is not an integer value");
+        return switch (n) {
+            case Byte b -> BigInteger.valueOf(b.longValue());
+            case Short s -> BigInteger.valueOf(s.longValue());
+            case Integer i -> BigInteger.valueOf(i.longValue());
+            case Long l -> BigInteger.valueOf(l.longValue());
+            case Float f -> {
+                double d = f.doubleValue();
+                if (d != Math.round(d)) {
+                    throw new IllegalArgumentException(f + " is not an integer value");
+                }
+                yield BigInteger.valueOf(f.longValue());
             }
-            return BigInteger.valueOf(n.longValue());
-        } else if (n instanceof BigDecimal) {
-            return ((BigDecimal) n).toBigIntegerExact();
-        } else if (n instanceof BigInteger) {
-            return ((BigInteger) n);
-        } else {
-            throw new IllegalArgumentException("Cannot convert [" + n + "] of class [" + n.getClass().getName() + "] to a BigInteger");
-        }
+            case Double d -> {
+                if (d != Math.round(d)) {
+                    throw new IllegalArgumentException(d + " is not an integer value");
+                }
+                yield BigInteger.valueOf(d.longValue());
+            }
+            case BigDecimal bd -> bd.toBigIntegerExact();
+            case BigInteger bi -> bi;
+            default -> throw new IllegalArgumentException(
+                "Cannot convert [" + n + "] of class [" + n.getClass().getName() + "] to a BigInteger"
+            );
+        };
     }
 
     /** Return the unsigned long (as {@link BigInteger}) that {@code n} stores, or throws an exception if the

libs/core/licenses/log4j-api-2.21.0.jar.sha1

@@ -0,0 +1 @@
+c7acbbd6f110cd1457c628da46245e355b1ee57a

libs/core/licenses/log4j-api-2.25.1.jar.sha1

@@ -0,0 +1 @@
+52f4ab3e4f5a80d52183842b7c81513f89f09f91