Merge branch 'main' into c3

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

Author: sandeshkr419

.idea/runConfigurations/Debug_OpenSearch.xml

@@ -5,7 +5,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased 3.x]
 ### Added
+- [Feature Request] Enhance Terms lookup query to support query clause instead of docId ([#18195](https://github.com/opensearch-project/OpenSearch/issues/18195))
 - Add hierarchical routing processors for ingest and search pipelines ([#18826](https://github.com/opensearch-project/OpenSearch/pull/18826))
+- Add ACL-aware routing processors for ingest and search pipelines ([#18834](https://github.com/opensearch-project/OpenSearch/pull/18834))
 - Add support for Warm Indices Write Block on Flood Watermark breach ([#18375](https://github.com/opensearch-project/OpenSearch/pull/18375))
 - FS stats for warm nodes based on addressable space ([#18767](https://github.com/opensearch-project/OpenSearch/pull/18767))
 - Add support for custom index name resolver from cluster plugin ([#18593](https://github.com/opensearch-project/OpenSearch/pull/18593))
@@ -30,6 +32,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Extend Approximation Framework to other numeric types ([#18530](https://github.com/opensearch-project/OpenSearch/issues/18530))
 - Add Semantic Version field type mapper and extensive unit tests([#18454](https://github.com/opensearch-project/OpenSearch/pull/18454))
 - Pass index settings to system ingest processor factories. ([#18708](https://github.com/opensearch-project/OpenSearch/pull/18708))
+- Add fetch phase profiling. ([#18664](https://github.com/opensearch-project/OpenSearch/pull/18664))
 - Include named queries from rescore contexts in matched_queries array ([#18697](https://github.com/opensearch-project/OpenSearch/pull/18697))
 - Add the configurable limit on rule cardinality ([#18663](https://github.com/opensearch-project/OpenSearch/pull/18663))
 - Disable approximation framework when dealing with multiple sorts ([#18763](https://github.com/opensearch-project/OpenSearch/pull/18763))
@@ -40,6 +43,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Make GRPC transport extensible to allow plugins to register and expose their own GRPC services ([#18516](https://github.com/opensearch-project/OpenSearch/pull/18516))
 - Added approximation support for range queries with now in date field ([#18511](https://github.com/opensearch-project/OpenSearch/pull/18511))
 - Upgrade to protobufs 0.6.0 and clean up deprecated TermQueryProtoUtils code ([#18880](https://github.com/opensearch-project/OpenSearch/pull/18880))
+- Expand fetch phase profiling to multi-shard queries ([#18887](https://github.com/opensearch-project/OpenSearch/pull/18887))
+- Prevent shard initialization failure due to streaming consumer errors ([#18877](https://github.com/opensearch-project/OpenSearch/pull/18877))
+- APIs for stream transport and new stream-based search api action ([#18722](https://github.com/opensearch-project/OpenSearch/pull/18722))
+- Added the core process for warming merged segments in remote-store enabled domains ([#18683](https://github.com/opensearch-project/OpenSearch/pull/18683))
 - Optimize Composite Aggregations by removing unnecessary object allocations ([#18531](https://github.com/opensearch-project/OpenSearch/pull/18531))
 
 ### Changed
@@ -48,6 +55,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Make node duress values cacheable ([#18649](https://github.com/opensearch-project/OpenSearch/pull/18649))
 - Change default value of remote_data_ratio, which is used in Searchable Snapshots and Writeable Warm from 0 to 5 and min allowed value to 1 ([#18767](https://github.com/opensearch-project/OpenSearch/pull/18767))
 - Making multi rate limiters in repository dynamic [#18069](https://github.com/opensearch-project/OpenSearch/pull/18069)
+- Optimize grouping for segment concurrent search by ensuring that documents within each group are as equal as possible ([#18451](https://github.com/opensearch-project/OpenSearch/pull/18451))
+- Move transport-grpc from a core plugin to a module ([#18897](https://github.com/opensearch-project/OpenSearch/pull/18897))
+- Remove `experimental` designation from transport-grpc settings ([#18915](https://github.com/opensearch-project/OpenSearch/pull/18915))
+- Rename package org.opensearch.plugin,transport.grpc to org.opensearch.transport.grpc ([#18923](https://github.com/opensearch-project/OpenSearch/pull/18923))
 
 ### Dependencies
 - Bump `stefanzweifel/git-auto-commit-action` from 5 to 6 ([#18524](https://github.com/opensearch-project/OpenSearch/pull/18524))
@@ -90,8 +101,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Field-level ignore_malformed should override index-level setting ([#18706](https://github.com/opensearch-project/OpenSearch/pull/18706))
 - Fixed Staggered merge -  load average replace with AverageTrackers, some Default thresholds modified ([#18666](https://github.com/opensearch-project/OpenSearch/pull/18666))
 - Use `new SecureRandom()` to avoid blocking ([18729](https://github.com/opensearch-project/OpenSearch/issues/18729))
+- Ignore archived settings on update ([#8714](https://github.com/opensearch-project/OpenSearch/issues/8714))
+- Ignore awareness attributes when a custom preference string is included with a search request ([#18848](https://github.com/opensearch-project/OpenSearch/pull/18848))
 - Use ScoreDoc instead of FieldDoc when creating TopScoreDocCollectorManager to avoid unnecessary conversion ([#18802](https://github.com/opensearch-project/OpenSearch/pull/18802))
 - Fix leafSorter optimization for ReadOnlyEngine and NRTReplicationEngine ([#18639](https://github.com/opensearch-project/OpenSearch/pull/18639))
+- Close IndexFieldDataService asynchronously ([#18888](https://github.com/opensearch-project/OpenSearch/pull/18888))
+- Fix query string regex queries incorrectly swallowing TooComplexToDeterminizeException ([#18883](https://github.com/opensearch-project/OpenSearch/pull/18883))
+- Fix socks5 user password settings for Azure repo ([#18904](https://github.com/opensearch-project/OpenSearch/pull/18904))
+- Reset isPipelineResolved to false to resolve the system ingest pipeline again. ([#18911](https://github.com/opensearch-project/OpenSearch/pull/18911))
 
 ### Security
 

CHANGELOG.md

@@ -5,8 +5,6 @@
 [![Code Coverage](https://codecov.io/gh/opensearch-project/OpenSearch/branch/main/graph/badge.svg)](https://codecov.io/gh/opensearch-project/OpenSearch)
 ![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/opensearch-project/OpenSearch?sort=semver)
 
-
-
 - [Welcome!](#welcome)
 - [Project Resources](#project-resources)
 - [Code of Conduct](#code-of-conduct)
@@ -17,7 +15,7 @@
 
 ## Welcome!
 
-**OpenSearch** is [a community-driven, open source fork](https://aws.amazon.com/blogs/opensource/introducing-opensearch/) of [Elasticsearch](https://en.wikipedia.org/wiki/Elasticsearch) and [Kibana](https://en.wikipedia.org/wiki/Kibana) following the [license change](https://blog.opensource.org/the-sspl-is-not-an-open-source-license/) in early 2021. We're looking to sustain (and evolve!) a search and analytics suite for the multitude of businesses who are dependent on the rights granted by the original, [Apache v2.0 License](LICENSE.txt).
+OpenSearch is an open-source, enterprise-grade search and observability suite that brings order to unstructured data at scale.
 
 ## Project Resources
 
@@ -49,7 +47,7 @@ Copyright OpenSearch Contributors. See [NOTICE](NOTICE.txt) for details.
 
 ## Trademark
 
-OpenSearch is a registered trademark of Amazon Web Services.
+OpenSearch is a registered trademark of LF Projects, LLC.
 
 OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.
 

README.md

@@ -32,6 +32,7 @@
 
 package org.opensearch.common.recycler;
 
+import org.opensearch.common.annotation.ExperimentalApi;
 import org.opensearch.common.lease.Releasable;
 
 /**
@@ -40,6 +41,7 @@
  *
  * @opensearch.internal
  */
+@ExperimentalApi
 public interface Recycler<T> {
 
     /**
@@ -73,6 +75,7 @@ interface C<T> {
      *
      * @opensearch.internal
      */
+    @ExperimentalApi
     interface V<T> extends Releasable {
 
         /** Reference to the value. */

libs/common/src/main/java/org/opensearch/common/recycler/Recycler.java

@@ -0,0 +1,248 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.ingest.common;
+
+import org.opensearch.action.admin.indices.create.CreateIndexRequest;
+import org.opensearch.action.get.GetResponse;
+import org.opensearch.action.index.IndexRequest;
+import org.opensearch.action.search.SearchResponse;
+import org.opensearch.common.hash.MurmurHash3;
+import org.opensearch.common.settings.Settings;
+import org.opensearch.core.common.bytes.BytesReference;
+import org.opensearch.core.xcontent.MediaTypeRegistry;
+import org.opensearch.index.query.TermQueryBuilder;
+import org.opensearch.plugins.Plugin;
+import org.opensearch.search.SearchHit;
+import org.opensearch.search.builder.SearchSourceBuilder;
+import org.opensearch.test.OpenSearchIntegTestCase;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Collection;
+import java.util.Map;
+
+import static org.opensearch.common.xcontent.XContentFactory.jsonBuilder;
+import static org.hamcrest.Matchers.equalTo;
+
+@OpenSearchIntegTestCase.ClusterScope(scope = OpenSearchIntegTestCase.Scope.TEST)
+public class AclRoutingProcessorIT extends OpenSearchIntegTestCase {
+
+    private static final Base64.Encoder BASE64_ENCODER = Base64.getUrlEncoder().withoutPadding();
+
+    @Override
+    protected Collection<Class<? extends Plugin>> nodePlugins() {
+        return Arrays.asList(IngestCommonModulePlugin.class);
+    }
+
+    public void testAclRoutingProcessor() throws Exception {
+        // Create ingest pipeline with ACL routing processor
+        String pipelineId = "acl-routing-test";
+        BytesReference pipelineConfig = BytesReference.bytes(
+            jsonBuilder().startObject()
+                .startArray("processors")
+                .startObject()
+                .startObject("acl_routing")
+                .field("acl_field", "team")
+                .field("target_field", "_routing")
+                .endObject()
+                .endObject()
+                .endArray()
+                .endObject()
+        );
+
+        client().admin().cluster().preparePutPipeline(pipelineId, pipelineConfig, MediaTypeRegistry.JSON).get();
+
+        // Create index with multiple shards - don't set default pipeline, use explicit pipeline parameter
+        String indexName = "test-acl-routing";
+        CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName).settings(
+            Settings.builder().put("number_of_shards", 3).put("number_of_replicas", 0).build()
+        )
+            .mapping(
+                jsonBuilder().startObject()
+                    .startObject("properties")
+                    .startObject("team")
+                    .field("type", "keyword")
+                    .endObject()
+                    .startObject("content")
+                    .field("type", "text")
+                    .endObject()
+                    .endObject()
+                    .endObject()
+            );
+
+        client().admin().indices().create(createIndexRequest).get();
+
+        // Index documents with explicit pipeline parameter
+        client().index(
+            new IndexRequest(indexName).id("1")
+                .source(jsonBuilder().startObject().field("team", "team-alpha").field("content", "Alpha content 1").endObject())
+                .setPipeline(pipelineId)
+        ).get();
+
+        client().index(
+            new IndexRequest(indexName).id("2")
+                .source(jsonBuilder().startObject().field("team", "team-alpha").field("content", "Alpha content 2").endObject())
+                .setPipeline(pipelineId)
+        ).get();
+
+        client().index(
+            new IndexRequest(indexName).id("3")
+                .source(jsonBuilder().startObject().field("team", "team-beta").field("content", "Beta content").endObject())
+                .setPipeline(pipelineId)
+        ).get();
+
+        // Refresh to make documents searchable
+        client().admin().indices().prepareRefresh(indexName).get();
+
+        // Test search functionality - documents should be searchable
+        SearchResponse searchResponse = client().prepareSearch(indexName)
+            .setSource(new SearchSourceBuilder().query(new TermQueryBuilder("team", "team-alpha")))
+            .get();
+
+        assertThat("Should find alpha team documents", searchResponse.getHits().getTotalHits().value(), equalTo(2L));
+
+        for (SearchHit hit : searchResponse.getHits().getHits()) {
+            String team = (String) hit.getSourceAsMap().get("team");
+            assertEquals("Found document should be from team alpha", "team-alpha", team);
+        }
+    }
+
+    public void testAclRoutingWithIgnoreMissing() throws Exception {
+        // Create pipeline with ignore_missing = true
+        String pipelineId = "acl-routing-ignore-missing";
+        BytesReference pipelineConfig = BytesReference.bytes(
+            jsonBuilder().startObject()
+                .startArray("processors")
+                .startObject()
+                .startObject("acl_routing")
+                .field("acl_field", "nonexistent_field")
+                .field("target_field", "_routing")
+                .field("ignore_missing", true)
+                .endObject()
+                .endObject()
+                .endArray()
+                .endObject()
+        );
+
+        client().admin().cluster().preparePutPipeline(pipelineId, pipelineConfig, MediaTypeRegistry.JSON).get();
+
+        String indexName = "test-ignore-missing";
+        CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName).settings(
+            Settings.builder().put("number_of_shards", 2).put("number_of_replicas", 0).put("index.default_pipeline", pipelineId).build()
+        );
+
+        client().admin().indices().create(createIndexRequest).get();
+
+        // Index document without the ACL field
+        IndexRequest indexRequest = new IndexRequest(indexName).id("missing1")
+            .source(
+                jsonBuilder().startObject().field("other_field", "some value").field("content", "Document without ACL field").endObject()
+            )
+            .setPipeline(pipelineId);
+
+        client().index(indexRequest).get();
+        client().admin().indices().prepareRefresh(indexName).get();
+
+        // Document should be indexed without routing since field was missing and ignored
+        GetResponse doc = client().prepareGet(indexName, "missing1").get();
+        assertTrue("Document should be indexed even with missing ACL field", doc.isExists());
+    }
+
+    public void testAclRoutingWithCustomTargetField() throws Exception {
+        // Create pipeline with custom target field
+        String pipelineId = "acl-routing-custom-target";
+        BytesReference pipelineConfig = BytesReference.bytes(
+            jsonBuilder().startObject()
+                .startArray("processors")
+                .startObject()
+                .startObject("acl_routing")
+                .field("acl_field", "department")
+                .field("target_field", "custom_routing")
+                .endObject()
+                .endObject()
+                .endArray()
+                .endObject()
+        );
+
+        client().admin().cluster().preparePutPipeline(pipelineId, pipelineConfig, MediaTypeRegistry.JSON).get();
+
+        String indexName = "test-custom-target";
+        CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName).settings(
+            Settings.builder().put("number_of_shards", 2).put("number_of_replicas", 0).put("index.default_pipeline", pipelineId).build()
+        );
+
+        client().admin().indices().create(createIndexRequest).get();
+
+        // Index document
+        IndexRequest indexRequest = new IndexRequest(indexName).id("custom1")
+            .source(jsonBuilder().startObject().field("department", "engineering").field("content", "Engineering document").endObject())
+            .setPipeline(pipelineId);
+
+        client().index(indexRequest).get();
+        client().admin().indices().prepareRefresh(indexName).get();
+
+        GetResponse doc = client().prepareGet(indexName, "custom1").get();
+        assertTrue("Document should exist", doc.isExists());
+
+        // Check that custom routing field was set
+        Map<String, Object> source = doc.getSource();
+        assertNotNull("Custom routing field should be set", source.get("custom_routing"));
+        assertEquals("Custom routing should match expected value", generateRoutingValue("engineering"), source.get("custom_routing"));
+    }
+
+    public void testAclRoutingProcessorRegistration() throws Exception {
+        // Verify processor is registered by attempting to create a pipeline
+        String pipelineId = "test-acl-processor-registration";
+        BytesReference pipelineConfig = BytesReference.bytes(
+            jsonBuilder().startObject()
+                .startArray("processors")
+                .startObject()
+                .startObject("acl_routing")
+                .field("acl_field", "team")
+                .endObject()
+                .endObject()
+                .endArray()
+                .endObject()
+        );
+
+        // This should succeed if processor is properly registered
+        client().admin().cluster().preparePutPipeline(pipelineId, pipelineConfig, MediaTypeRegistry.JSON).get();
+
+        // Verify pipeline was created
+        var getPipelineResponse = client().admin().cluster().prepareGetPipeline(pipelineId).get();
+        assertTrue("Pipeline should be created successfully", getPipelineResponse.isFound());
+
+        // Clean up
+        client().admin().cluster().prepareDeletePipeline(pipelineId).get();
+    }
+
+    // Helper method to generate routing value (mirrors processor logic)
+    private String generateRoutingValue(String aclValue) {
+        // Use MurmurHash3 for consistent hashing (same as processor)
+        byte[] bytes = aclValue.getBytes(StandardCharsets.UTF_8);
+        MurmurHash3.Hash128 hash = MurmurHash3.hash128(bytes, 0, bytes.length, 0, new MurmurHash3.Hash128());
+
+        // Convert to base64 for routing value
+        byte[] hashBytes = new byte[16];
+        System.arraycopy(longToBytes(hash.h1), 0, hashBytes, 0, 8);
+        System.arraycopy(longToBytes(hash.h2), 0, hashBytes, 8, 8);
+
+        return BASE64_ENCODER.encodeToString(hashBytes);
+    }
+
+    private byte[] longToBytes(long value) {
+        byte[] result = new byte[8];
+        for (int i = 7; i >= 0; i--) {
+            result[i] = (byte) (value & 0xFF);
+            value >>= 8;
+        }
+        return result;
+    }
+}