Return 409 Conflict HTTP status instead of 500 on failure to concurre…

…ntly execute snapshots

Signed-off-by: Mani <singh.mani1231@gmail.com>

.github/workflows/check-compatibility.yml

@@ -0,0 +1,35 @@
+---
+name: Check Compatibility
+
+on:
+  pull_request_target
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Run compatibility task
+        run: ./gradlew checkCompatibility | tee $HOME/gradlew-check.out
+
+      - name: Get results
+        run: |
+          echo 'Compatibility status:' > ${{ github.workspace }}/results.txt && echo '```' >> ${{ github.workspace }}/results.txt
+          grep -e 'Compatible components' -e 'Incompatible components' -e 'Components skipped' -A 2 -B 3 $HOME/gradlew-check.out >> "${{ github.workspace }}/results.txt"
+          echo '```' >> ${{ github.workspace }}/results.txt
+
+      - name: GitHub App token
+        id: github_app_token
+        uses: tibdex/github-app-token@v1.6.0
+        with:
+          app_id: ${{ secrets.APP_ID }}
+          private_key: ${{ secrets.APP_PRIVATE_KEY }}
+          installation_id: 22958780
+
+      - name: Add comment on the PR
+        uses: peter-evans/create-or-update-comment@v3
+        with:
+          token: ${{ steps.github_app_token.outputs.token }}
+          issue-number: ${{ github.event.number }}
+          body-path: "${{ github.workspace }}/results.txt"

CHANGELOG.md

@@ -48,7 +48,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Remote Segment Store Repository setting moved from `index.remote_store.repository` to `index.remote_store.segment.repository` and `cluster.remote_store.repository` to `cluster.remote_store.segment.repository` respectively for Index and Cluster level settings ([#8719](https://github.com/opensearch-project/OpenSearch/pull/8719))
 - [Remote Store] Add support to restore only unassigned shards of an index ([#8792](https://github.com/opensearch-project/OpenSearch/pull/8792))
 - Replace the deprecated IndexReader APIs with new storedFields() & termVectors() ([#7792](https://github.com/opensearch-project/OpenSearch/pull/7792))
-- Return 409 Conflict HTTP status instead of 500 on failure to concurrently execute snapshots ([#5855](https://github.com/opensearch-project/OpenSearch/pull/5855))
+- [Remote Store] Restrict user override for remote store index level settings ([#8812](https://github.com/opensearch-project/OpenSearch/pull/8812))
+- Return 409 Conflict HTTP status instead of 503 on failure to concurrently execute snapshots ([#8986](https://github.com/opensearch-project/OpenSearch/pull/5855))
 
 ### Deprecated
 
@@ -81,6 +82,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Start replication checkpointTimers on primary before segments upload to remote store. ([#8221]()https://github.com/opensearch-project/OpenSearch/pull/8221)
 - [distribution/archives] [Linux] [x64] Provide the variant of the distributions bundled with JRE ([#8195]()https://github.com/opensearch-project/OpenSearch/pull/8195)
 - Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache ([#8606](https://github.com/opensearch-project/OpenSearch/pull/8606))
+- Disallow compression level to be set for default and best_compression index codecs ([#8737]()https://github.com/opensearch-project/OpenSearch/pull/8737)
 
 ### Dependencies
 - Bump `org.apache.logging.log4j:log4j-core` from 2.17.1 to 2.20.0 ([#8307](https://github.com/opensearch-project/OpenSearch/pull/8307))
@@ -93,6 +95,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.google.http-client:google-http-client-gson` from 1.43.2 to 1.43.3 ([#8840](https://github.com/opensearch-project/OpenSearch/pull/8840))
 - OpenJDK Update (July 2023 Patch releases) ([#8868](https://github.com/opensearch-project/OpenSearch/pull/8868)
 - Bump `hadoop` libraries from 3.3.4 to 3.3.6 ([#6995](https://github.com/opensearch-project/OpenSearch/pull/6995))
+- Bump `com.gradle.enterprise` from 3.13.3 to 3.14.1 ([#8996](https://github.com/opensearch-project/OpenSearch/pull/8996))
+- Bump `org.apache.commons:commons-lang3` from 3.12.0 to 3.13.0 ([#8995](https://github.com/opensearch-project/OpenSearch/pull/8995))
+- Bump `com.google.cloud:google-cloud-core-http` from 2.21.0 to 2.21.1 ([#8999](https://github.com/opensearch-project/OpenSearch/pull/8999))
+- Bump `com.maxmind.geoip2:geoip2` from 4.0.1 to 4.1.0 ([#8998](https://github.com/opensearch-project/OpenSearch/pull/8998))
+- Bump `org.apache.commons:commons-lang3` from 3.12.0 to 3.13.0 in /plugins/repository-hdfs ([#8997](https://github.com/opensearch-project/OpenSearch/pull/8997))
+- Bump `netty` from 4.1.94.Final to 4.1.96.Final ([#9030](https://github.com/opensearch-project/OpenSearch/pull/9030))
 
 ### Changed
 - Perform aggregation postCollection in ContextIndexSearcher after searching leaves ([#8303](https://github.com/opensearch-project/OpenSearch/pull/8303))
@@ -101,12 +109,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Exclude 'benchmarks' from codecov report ([#8805](https://github.com/opensearch-project/OpenSearch/pull/8805))
 - [Refactor] MediaTypeParser to MediaTypeParserRegistry ([#8636](https://github.com/opensearch-project/OpenSearch/pull/8636))
 - Create separate SourceLookup instance per segment slice in SignificantTextAggregatorFactory ([#8807](https://github.com/opensearch-project/OpenSearch/pull/8807))
+- Add support for aggregation profiler with concurrent aggregation ([#8801](https://github.com/opensearch-project/OpenSearch/pull/8801))
+- [Remove] Deprecated Fractional ByteSizeValue support #9005 ([#9005](https://github.com/opensearch-project/OpenSearch/pull/9005))
 
 ### Deprecated
 
 ### Removed
 
 ### Fixed
+- Fix flaky ResourceAwareTasksTests.testBasicTaskResourceTracking test ([#8993](https://github.com/opensearch-project/OpenSearch/pull/8993))
 
 ### Security
 

benchmarks/src/main/java/org/opensearch/benchmark/routing/allocation/Allocators.java

@@ -45,7 +45,7 @@
 import org.opensearch.cluster.routing.allocation.decider.AllocationDeciders;
 import org.opensearch.common.settings.ClusterSettings;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.transport.TransportAddress;
+import org.opensearch.core.common.transport.TransportAddress;
 import org.opensearch.common.util.set.Sets;
 import org.opensearch.gateway.GatewayAllocator;
 import org.opensearch.snapshots.EmptySnapshotsInfoService;

benchmarks/src/main/java/org/opensearch/benchmark/search/aggregations/TermsReduceBenchmark.java

@@ -40,14 +40,14 @@
 import org.opensearch.action.search.SearchPhaseController;
 import org.opensearch.action.search.SearchProgressListener;
 import org.opensearch.action.search.SearchRequest;
-import org.opensearch.common.breaker.CircuitBreaker;
-import org.opensearch.common.breaker.NoopCircuitBreaker;
+import org.opensearch.core.common.breaker.CircuitBreaker;
+import org.opensearch.core.common.breaker.NoopCircuitBreaker;
 import org.opensearch.core.common.io.stream.NamedWriteableRegistry;
 import org.opensearch.common.lucene.search.TopDocsAndMaxScore;
 import org.opensearch.common.settings.Settings;
 import org.opensearch.core.index.Index;
 import org.opensearch.core.index.shard.ShardId;
-import org.opensearch.indices.breaker.NoneCircuitBreakerService;
+import org.opensearch.core.indices.breaker.NoneCircuitBreakerService;
 import org.opensearch.search.DocValueFormat;
 import org.opensearch.search.SearchModule;
 import org.opensearch.search.SearchShardTarget;

benchmarks/src/main/java/org/opensearch/benchmark/store/remote/filecache/FileCacheBenchmark.java

@@ -27,8 +27,8 @@
 import org.openjdk.jmh.annotations.Threads;
 import org.openjdk.jmh.annotations.Warmup;
 import org.openjdk.jmh.infra.Blackhole;
-import org.opensearch.common.breaker.CircuitBreaker;
-import org.opensearch.common.breaker.NoopCircuitBreaker;
+import org.opensearch.core.common.breaker.CircuitBreaker;
+import org.opensearch.core.common.breaker.NoopCircuitBreaker;
 import org.opensearch.index.store.remote.filecache.CachedIndexInput;
 import org.opensearch.index.store.remote.filecache.FileCache;
 import org.opensearch.index.store.remote.filecache.FileCacheFactory;

build.gradle

@@ -487,7 +487,6 @@ subprojects {
         includeClasses.add("org.opensearch.cluster.metadata.IndexGraveyardTests")
         includeClasses.add("org.opensearch.cluster.routing.MovePrimaryFirstTests")
         includeClasses.add("org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT")
-        includeClasses.add("org.opensearch.cluster.service.MasterServiceTests")
         includeClasses.add("org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests")
         includeClasses.add("org.opensearch.gateway.RecoveryFromGatewayIT")
         includeClasses.add("org.opensearch.gateway.ReplicaShardAllocatorIT")

buildSrc/src/main/groovy/org/opensearch/gradle/CheckCompatibilityTask.groovy

@@ -40,8 +40,8 @@ class CheckCompatibilityTask extends DefaultTask {
 
     @TaskAction
     void checkCompatibility() {
-        logger.info("Checking compatibility for: $repositoryUrls for $ref")
         repositoryUrls.parallelStream().forEach { repositoryUrl ->
+            logger.lifecycle("Checking compatibility for: $repositoryUrl with ref: $ref")
             def tempDir = File.createTempDir()
             try {
                 if (cloneAndCheckout(repositoryUrl, tempDir)) {
@@ -81,8 +81,16 @@ class CheckCompatibilityTask extends DefaultTask {
 
     protected static List getRepoUrls() {
         def json = new JsonSlurper().parse(REPO_URL.toURL())
-        def labels = json.projects.values()
-        return labels as List
+        def repository = json.projects.values()
+        def repoUrls = replaceSshWithHttps(repository as List)
+        return repoUrls
+    }
+
+    protected static replaceSshWithHttps(List<String> repoList) {
+        repoList.replaceAll { element ->
+            element.replace("git@github.com:", "https://github.com/")
+        }
+        return repoList
     }
 
     protected boolean cloneAndCheckout(repoUrl, directory) {

buildSrc/version.properties

@@ -28,7 +28,7 @@ jakarta_annotation = 1.3.5
 # when updating the JNA version, also update the version in buildSrc/build.gradle
 jna               = 5.5.0
 
-netty             = 4.1.94.Final
+netty             = 4.1.96.Final
 joda              = 2.12.2
 
 # client dependencies

client/rest-high-level/src/main/java/org/opensearch/client/indices/DataStreamsStatsResponse.java

@@ -34,7 +34,7 @@
 
 import org.opensearch.client.core.BroadcastResponse;
 import org.opensearch.core.ParseField;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.core.xcontent.ConstructingObjectParser;
 import org.opensearch.core.xcontent.ObjectParser;
 import org.opensearch.core.xcontent.XContentParser;

client/rest-high-level/src/main/java/org/opensearch/client/indices/ResizeRequest.java

@@ -39,7 +39,7 @@
 import org.opensearch.common.settings.Settings;
 import org.opensearch.core.xcontent.ToXContentObject;
 import org.opensearch.core.xcontent.XContentBuilder;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeValue;
 
 import java.io.IOException;
 import java.util.Collections;

client/rest-high-level/src/main/java/org/opensearch/client/indices/rollover/RolloverRequest.java

@@ -37,7 +37,7 @@
 import org.opensearch.action.admin.indices.rollover.MaxSizeCondition;
 import org.opensearch.client.TimedRequest;
 import org.opensearch.client.indices.CreateIndexRequest;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.core.xcontent.ToXContentObject;
 import org.opensearch.core.xcontent.XContentBuilder;

client/rest-high-level/src/test/java/org/opensearch/client/BulkProcessorIT.java

@@ -44,8 +44,8 @@
 import org.opensearch.action.search.SearchRequest;
 import org.opensearch.common.Strings;
 import org.opensearch.core.common.bytes.BytesArray;
-import org.opensearch.common.unit.ByteSizeUnit;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.core.xcontent.XContentBuilder;
 import org.opensearch.common.xcontent.XContentType;

client/rest-high-level/src/test/java/org/opensearch/client/ClusterClientIT.java

@@ -61,7 +61,7 @@
 import org.opensearch.cluster.routing.allocation.decider.EnableAllocationDecider;
 import org.opensearch.common.compress.CompressedXContent;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeUnit;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.common.xcontent.XContentType;
 import org.opensearch.common.xcontent.support.XContentMapValues;

client/rest-high-level/src/test/java/org/opensearch/client/CrudIT.java

@@ -60,8 +60,8 @@
 import org.opensearch.client.indices.GetIndexRequest;
 import org.opensearch.core.common.bytes.BytesReference;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.unit.ByteSizeUnit;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.core.common.Strings;
 import org.opensearch.core.xcontent.XContentBuilder;
 import org.opensearch.common.xcontent.XContentFactory;

client/rest-high-level/src/test/java/org/opensearch/client/IndicesClientIT.java

@@ -112,8 +112,8 @@
 import org.opensearch.common.compress.CompressedXContent;
 import org.opensearch.common.settings.Setting;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.unit.ByteSizeUnit;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.core.common.Strings;
 import org.opensearch.core.xcontent.XContentBuilder;

client/rest-high-level/src/test/java/org/opensearch/client/IndicesRequestConvertersTests.java

@@ -79,7 +79,7 @@
 import org.opensearch.core.common.Strings;
 import org.opensearch.test.OpenSearchTestCase;
 import org.junit.Assert;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeValue;
 
 import java.io.IOException;
 import java.util.Arrays;

client/rest-high-level/src/test/java/org/opensearch/client/SnapshotRequestConvertersTests.java

@@ -48,7 +48,7 @@
 import org.opensearch.action.support.master.AcknowledgedRequest;
 import org.opensearch.common.io.PathUtils;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeUnit;
 import org.opensearch.repositories.fs.FsRepository;
 import org.opensearch.test.OpenSearchTestCase;
 

client/rest-high-level/src/test/java/org/opensearch/client/documentation/CRUDDocumentationIT.java

@@ -77,8 +77,8 @@
 import org.opensearch.common.Strings;
 import org.opensearch.core.common.bytes.BytesArray;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.unit.ByteSizeUnit;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.core.xcontent.XContentBuilder;
 import org.opensearch.common.xcontent.XContentFactory;

client/rest-high-level/src/test/java/org/opensearch/client/documentation/ClusterClientDocumentationIT.java

@@ -63,7 +63,7 @@
 import org.opensearch.common.Priority;
 import org.opensearch.common.compress.CompressedXContent;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeUnit;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.common.xcontent.XContentType;
 import org.opensearch.indices.recovery.RecoverySettings;

client/rest-high-level/src/test/java/org/opensearch/client/documentation/IndicesClientDocumentationIT.java

@@ -104,8 +104,8 @@
 import org.opensearch.cluster.metadata.Template;
 import org.opensearch.common.compress.CompressedXContent;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.common.unit.ByteSizeUnit;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeUnit;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.core.xcontent.XContentBuilder;
 import org.opensearch.common.xcontent.XContentFactory;

client/rest-high-level/src/test/java/org/opensearch/client/indices/DataStreamsStatsResponseTests.java

@@ -36,7 +36,7 @@
 import org.opensearch.action.admin.indices.datastream.DataStreamsStatsAction;
 import org.opensearch.core.action.support.DefaultShardOperationFailedException;
 import org.opensearch.client.AbstractResponseTestCase;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.core.xcontent.XContentParser;
 import org.opensearch.common.xcontent.XContentType;
 

client/rest-high-level/src/test/java/org/opensearch/client/indices/rollover/RolloverRequestTests.java

@@ -36,7 +36,7 @@
 import org.opensearch.action.admin.indices.rollover.MaxAgeCondition;
 import org.opensearch.action.admin.indices.rollover.MaxDocsCondition;
 import org.opensearch.action.admin.indices.rollover.MaxSizeCondition;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.test.OpenSearchTestCase;
 

client/rest-high-level/src/test/java/org/opensearch/client/indices/rollover/RolloverResponseTests.java

@@ -36,7 +36,7 @@
 import org.opensearch.action.admin.indices.rollover.MaxAgeCondition;
 import org.opensearch.action.admin.indices.rollover.MaxDocsCondition;
 import org.opensearch.action.admin.indices.rollover.MaxSizeCondition;
-import org.opensearch.common.unit.ByteSizeValue;
+import org.opensearch.core.common.unit.ByteSizeValue;
 import org.opensearch.common.unit.TimeValue;
 import org.opensearch.core.xcontent.XContentBuilder;
 import org.opensearch.test.OpenSearchTestCase;

client/rest-high-level/src/test/java/org/opensearch/client/tasks/CancelTasksResponseTests.java

@@ -40,7 +40,7 @@
 import org.opensearch.cluster.node.DiscoveryNodes;
 import org.opensearch.core.common.io.stream.StreamInput;
 import org.opensearch.core.common.io.stream.StreamOutput;
-import org.opensearch.common.transport.TransportAddress;
+import org.opensearch.core.common.transport.TransportAddress;
 import org.opensearch.core.xcontent.XContentBuilder;
 import org.opensearch.core.xcontent.XContentParser;
 import org.opensearch.common.xcontent.XContentType;

server/src/main/java/org/opensearch/common/component/AbstractLifecycleComponent.java → libs/common/src/main/java/org/opensearch/common/lifecycle/AbstractLifecycleComponent.java

@@ -30,7 +30,7 @@
  * GitHub history for details.
  */
 
-package org.opensearch.common.component;
+package org.opensearch.common.lifecycle;
 
 import java.io.IOException;
 import java.io.UncheckedIOException;

server/src/main/java/org/opensearch/common/component/Lifecycle.java → libs/common/src/main/java/org/opensearch/common/lifecycle/Lifecycle.java

@@ -30,7 +30,7 @@
  * GitHub history for details.
  */
 
-package org.opensearch.common.component;
+package org.opensearch.common.lifecycle;
 
 /**
  * Lifecycle state. Allows the following transitions:

server/src/main/java/org/opensearch/common/component/LifecycleComponent.java → libs/common/src/main/java/org/opensearch/common/lifecycle/LifecycleComponent.java

@@ -30,7 +30,7 @@
  * GitHub history for details.
  */
 
-package org.opensearch.common.component;
+package org.opensearch.common.lifecycle;
 
 import org.opensearch.common.lease.Releasable;
 

server/src/main/java/org/opensearch/common/component/LifecycleListener.java → libs/common/src/main/java/org/opensearch/common/lifecycle/LifecycleListener.java

@@ -30,7 +30,7 @@
  * GitHub history for details.
  */
 
-package org.opensearch.common.component;
+package org.opensearch.common.lifecycle;
 
 /**
  * Base lifecycle listener.

libs/common/src/main/java/org/opensearch/common/lifecycle/package-info.java

@@ -0,0 +1,16 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+/**
+ * Foundation implementation for a object lifecycle.
+ *
+ * See {@link org.opensearch.common.lifecycle.Lifecycle} for example usage
+ *
+ * @opensearch.internal
+ */
+package org.opensearch.common.lifecycle;

libs/common/src/main/java/org/opensearch/common/network/package-info.java

@@ -0,0 +1,10 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+/** common network layer classes used across the code base */
+package org.opensearch.common.network;