Merge branch '2.x' of https://github.com/opensearch-project/OpenSearch …

…into pits2.x

.ci/bwcVersions

@@ -42,6 +42,7 @@ BWC_VERSION:
  - "1.3.3"
  - "1.3.4"
  - "1.3.5"
+ - "1.3.6"
  - "2.0.0"
  - "2.0.1"
  - "2.0.2"
@@ -50,3 +51,5 @@ BWC_VERSION:
  - "2.2.0"
  - "2.2.1"
  - "2.2.2"
+ - "2.3.0"
+ - "2.3.1"

.github/workflows/changelog_verifier.yml

@@ -14,3 +14,5 @@ jobs:
  ref: ${{ github.event.pull_request.head.sha }}
 
  - uses: dangoslen/changelog-enforcer@v3
+ with:
+ skipLabels: "autocut"

.github/workflows/version.yml

@@ -5,7 +5,7 @@ on:
  tags:
  - '*.*.*'
 
-jobs: 
+jobs:
  build:
  runs-on: ubuntu-latest
  steps:
@@ -60,6 +60,8 @@ jobs:
  branch: 'create-pull-request/patch-${{ env.BASE }}'
  commit-message: Incremented version to ${{ env.NEXT_VERSION }}
  delete-branch: true
+ labels: |
+ autocut
  title: '[AUTO] Incremented version to ${{ env.NEXT_VERSION }}.'
  body: |
  I've noticed that a new tag ${{ env.TAG }} was pushed, and incremented the version from ${{ env.CURRENT_VERSION }} to ${{ env.NEXT_VERSION }}.
@@ -84,6 +86,8 @@ jobs:
  branch: 'create-pull-request/patch-${{ env.BASE_X }}'
  commit-message: Added bwc version ${{ env.NEXT_VERSION }}
  delete-branch: true
+ labels: |
+ autocut
  title: '[AUTO] [${{ env.BASE_X }}] Added bwc version ${{ env.NEXT_VERSION }}.'
  body: |
  I've noticed that a new tag ${{ env.TAG }} was pushed, and added a bwc version ${{ env.NEXT_VERSION }}.
@@ -108,6 +112,8 @@ jobs:
  branch: 'create-pull-request/patch-main'
  commit-message: Added bwc version ${{ env.NEXT_VERSION }}
  delete-branch: true
+ labels: |
+ autocut
  title: '[AUTO] [main] Added bwc version ${{ env.NEXT_VERSION }}.'
  body: |
  I've noticed that a new tag ${{ env.TAG }} was pushed, and added a bwc version ${{ env.NEXT_VERSION }}.

CHANGELOG.md

@@ -7,12 +7,23 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Point in time rest layer changes for create and delete PIT API ([#4064](https://github.com/opensearch-project/OpenSearch/pull/4064))
 - Add failover support with Segment Replication enabled. ([#4325](https://github.com/opensearch-project/OpenSearch/pull/4325)
 - Added @dreamer-89 as an Opensearch maintainer ([#4342](https://github.com/opensearch-project/OpenSearch/pull/4342))
+- Added release notes for 1.3.5 ([#4343](https://github.com/opensearch-project/OpenSearch/pull/4343))
+- Added release notes for 2.2.1 ([#4344](https://github.com/opensearch-project/OpenSearch/pull/4344))
+- Label configuration for dependabot PRs ([#4348](https://github.com/opensearch-project/OpenSearch/pull/4348))
+- Support for HTTP/2 (server-side) ([#3847](https://github.com/opensearch-project/OpenSearch/pull/3847))
+- BWC version 2.2.2 ([#4383](https://github.com/opensearch-project/OpenSearch/pull/4383))
+- Support for labels on version bump PRs, skip label support for changelog verifier ([#4391](https://github.com/opensearch-project/OpenSearch/pull/4391))
+
+### Dependencies
+- Bumps `com.diffplug.spotless` from 6.9.1 to 6.10.0
+- Bumps `xmlbeans` from 5.1.0 to 5.1.1
 
 ### Changed
 - Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
-- Update to Netty 4.1.80.Final ([#4359](https://github.com/opensearch-project/OpenSearch/pull/4359))
 - Use RemoteSegmentStoreDirectory instead of RemoteDirectory ([#4240](https://github.com/opensearch-project/OpenSearch/pull/4240))
 - Add index specific setting for remote repository ([#4253](https://github.com/opensearch-project/OpenSearch/pull/4253))
+- [Segment Replication] Update replicas to commit SegmentInfos instead of relying on SIS files from primary shards. ([#4402](https://github.com/opensearch-project/OpenSearch/pull/4402))
+- Change the version to remove deprecated code of adding node name into log pattern of log4j property file ([#4569](https://github.com/opensearch-project/OpenSearch/pull/4569))
 
 ### Deprecated
 
@@ -28,6 +39,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Commit workflow for dependabot changelog helper ([#4331](https://github.com/opensearch-project/OpenSearch/pull/4331))
 - Fixed cancellation of segment replication events ([#4225](https://github.com/opensearch-project/OpenSearch/pull/4225))
 - Bugs for dependabot changelog verifier workflow ([#4364](https://github.com/opensearch-project/OpenSearch/pull/4364))
+- [Bug]: gradle check failing with java heap OutOfMemoryError (([#4328](https://github.com/opensearch-project/OpenSearch/
+- `opensearch.bat` fails to execute when install path includes spaces ([#4362](https://github.com/opensearch-project/OpenSearch/pull/4362))
+- Getting security exception due to access denied 'java.lang.RuntimePermission' 'accessDeclaredMembers' when trying to get snapshot with S3 IRSA ([#4469](https://github.com/opensearch-project/OpenSearch/pull/4469))
+- Fixed flaky test `ResourceAwareTasksTests.testTaskIdPersistsInThreadContext` ([#4484](https://github.com/opensearch-project/OpenSearch/pull/4484))
+- Fixed the ignore_malformed setting to also ignore objects ([#4494](https://github.com/opensearch-project/OpenSearch/pull/4494))
 - Fixed the `_cat/shards/10_basic.yml` test cases fix.
 
 ### Security
@@ -38,6 +54,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
 - Add timing data and more granular stages to SegmentReplicationState ([#4367](https://github.com/opensearch-project/OpenSearch/pull/4367))
 - BWC version 2.2.2 ([#4385](https://github.com/opensearch-project/OpenSearch/pull/4385))
+- BWC version 1.3.6 ([#4452](https://github.com/opensearch-project/OpenSearch/pull/4452))
+- Bump current version to 2.4.0 on 2.x branch ([#4454](https://github.com/opensearch-project/OpenSearch/pull/4454))
+- 2.3.0 release notes ([#4457](https://github.com/opensearch-project/OpenSearch/pull/4457))
+- Add BWC version 2.3.1 ([#4512](https://github.com/opensearch-project/OpenSearch/pull/4512))
+- Updated jackson to 2.13.4 and snakeyml to 1.32 ([#4563](https://github.com/opensearch-project/OpenSearch/pull/4563))
 - Added RestLayer Changes for PIT stats ([#4217](https://github.com/opensearch-project/OpenSearch/pull/4217))
 
 ### Changed
@@ -52,6 +73,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Fixed cancellation of segment replication events ([#4225](https://github.com/opensearch-project/OpenSearch/pull/4225))
 - [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica ([#4363](https://github.com/opensearch-project/OpenSearch/pull/4363))
 - [Segment Replication] Bump segment infos counter before commit during replica promotion ([#4365](https://github.com/opensearch-project/OpenSearch/pull/4365))
+- [Segment Replication] Update flaky testOnNewCheckpointFromNewPrimaryCancelOngoingReplication unit test ([#4414](https://github.com/opensearch-project/OpenSearch/pull/4414))
+- [Segment Replication] Extend FileChunkWriter to allow cancel on transport client ([#4386](https://github.com/opensearch-project/OpenSearch/pull/4386))
+- [Segment Replication] Fix NoSuchFileExceptions with segment replication when computing primary metadata snapshots ([#4366](https://github.com/opensearch-project/OpenSearch/pull/4366))
+- [Segment Replication] Fix timeout issue by calculating time needed to process getSegmentFiles ([#4434](https://github.com/opensearch-project/OpenSearch/pull/4434))
+- [Segment Replication] Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards.
 
 ### Security
 

build.gradle

@@ -264,6 +264,12 @@ tasks.register("branchConsistency") {
 allprojects {
  // configure compiler options
  tasks.withType(JavaCompile).configureEach { JavaCompile compile ->
+ options.fork = true
+
+ configure(options.forkOptions) {
+ memoryMaximumSize = project.property('options.forkOptions.memoryMaximumSize')
+ }
+
  // See please https://bugs.openjdk.java.net/browse/JDK-8209058
  if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_11) {
  compile.options.compilerArgs << '-Werror'
@@ -389,6 +395,10 @@ allprojects {
 // the dependency is added.
 gradle.projectsEvaluated {
  allprojects {
+ project.tasks.withType(JavaForkOptions) {
+ maxHeapSize project.property('options.forkOptions.memoryMaximumSize')
+ }
+
  if (project.path == ':test:framework') {
  // :test:framework:test cannot run before and after :server:test
  return

buildSrc/version.properties

@@ -1,4 +1,4 @@
-opensearch = 2.3.0
+opensearch = 2.4.0
 lucene = 9.3.0
 
 bundled_jdk_vendor = adoptium
@@ -9,9 +9,9 @@ bundled_jdk = 17.0.4+8
 # optional dependencies
 spatial4j = 0.7
 jts = 1.15.0
-jackson = 2.13.3
-jackson_databind = 2.13.3
-snakeyaml = 1.31
+jackson = 2.13.4
+jackson_databind = 2.13.4
+snakeyaml = 1.32
 icu4j = 70.1
 supercsv = 2.4.0
 log4j = 2.17.1
@@ -21,7 +21,7 @@ asm = 9.3
 # when updating the JNA version, also update the version in buildSrc/build.gradle
 jna = 5.5.0
 
-netty = 4.1.80.Final
+netty = 4.1.79.Final
 joda = 2.10.12
 
 # client dependencies

client/sniffer/licenses/jackson-core-2.13.4.jar.sha1

@@ -0,0 +1 @@
+0cf934c681294b97ef6d80082faeefbe1edadf56

distribution/src/bin/opensearch-cli.bat

@@ -16,7 +16,7 @@ rem use a small heap size for the CLI tools, and thus the serial collector to
 rem avoid stealing many CPU cycles; a user can override by setting OPENSEARCH_JAVA_OPTS
 set OPENSEARCH_JAVA_OPTS=-Xms4m -Xmx64m -XX:+UseSerialGC %OPENSEARCH_JAVA_OPTS%
 
-%JAVA% ^
+"%JAVA%" ^
  %OPENSEARCH_JAVA_OPTS% ^
  -Dopensearch.path.home="%OPENSEARCH_HOME%" ^
  -Dopensearch.path.conf="%OPENSEARCH_PATH_CONF%" ^

distribution/src/bin/opensearch-env.bat

@@ -43,14 +43,14 @@ rem comparing to empty string makes this equivalent to bash -v check on env var
 rem and allows to effectively force use of the bundled jdk when launching OpenSearch
 rem by setting OPENSEARCH_JAVA_HOME= and JAVA_HOME= 
 if not "%OPENSEARCH_JAVA_HOME%" == "" (
- set JAVA="%OPENSEARCH_JAVA_HOME%\bin\java.exe"
+ set "JAVA=%OPENSEARCH_JAVA_HOME%\bin\java.exe"
  set JAVA_TYPE=OPENSEARCH_JAVA_HOME 
 ) else if not "%JAVA_HOME%" == "" (
- set JAVA="%JAVA_HOME%\bin\java.exe"
+ set "JAVA=%JAVA_HOME%\bin\java.exe"
  set JAVA_TYPE=JAVA_HOME
 ) else (
- set JAVA="%OPENSEARCH_HOME%\jdk\bin\java.exe"
- set JAVA_HOME="%OPENSEARCH_HOME%\jdk"
+ set "JAVA=%OPENSEARCH_HOME%\jdk\bin\java.exe"
+ set "JAVA_HOME=%OPENSEARCH_HOME%\jdk"
  set JAVA_TYPE=bundled jdk
 )
 
@@ -73,4 +73,4 @@ if defined JAVA_OPTS (
 )
 
 rem check the Java version
-%JAVA% -cp "%OPENSEARCH_CLASSPATH%" "org.opensearch.tools.java_version_checker.JavaVersionChecker" || exit /b 1
+"%JAVA%" -cp "%OPENSEARCH_CLASSPATH%" "org.opensearch.tools.java_version_checker.JavaVersionChecker" || exit /b 1

distribution/src/bin/opensearch-service.bat

@@ -121,7 +121,7 @@ if exist "%JAVA_HOME%\bin\server\jvm.dll" (
 
 :foundJVM
 if not defined OPENSEARCH_TMPDIR (
- for /f "tokens=* usebackq" %%a in (`CALL %JAVA% -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.TempDirectory"`) do set OPENSEARCH_TMPDIR=%%a
+ for /f "tokens=* usebackq" %%a in (`CALL "%JAVA%" -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.TempDirectory"`) do set OPENSEARCH_TMPDIR=%%a
 )
 
 rem The JVM options parser produces the final JVM options to start
@@ -135,7 +135,7 @@ rem - third, JVM options from OPENSEARCH_JAVA_OPTS are applied
 rem - fourth, ergonomic JVM options are applied
 
 @setlocal
-for /F "usebackq delims=" %%a in (`CALL %JAVA% -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.JvmOptionsParser" "!OPENSEARCH_PATH_CONF!" ^|^| echo jvm_options_parser_failed`) do set OPENSEARCH_JAVA_OPTS=%%a
+for /F "usebackq delims=" %%a in (`CALL "%JAVA%" -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.JvmOptionsParser" "!OPENSEARCH_PATH_CONF!" ^|^| echo jvm_options_parser_failed`) do set OPENSEARCH_JAVA_OPTS=%%a
 @endlocal & set "MAYBE_JVM_OPTIONS_PARSER_FAILED=%OPENSEARCH_JAVA_OPTS%" & set OPENSEARCH_JAVA_OPTS=%OPENSEARCH_JAVA_OPTS%
 
 if "%MAYBE_JVM_OPTIONS_PARSER_FAILED%" == "jvm_options_parser_failed" (

distribution/src/bin/opensearch.bat

@@ -75,7 +75,7 @@ IF "%checkpassword%"=="Y" (
 )
 
 if not defined OPENSEARCH_TMPDIR (
- for /f "tokens=* usebackq" %%a in (`CALL %JAVA% -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.TempDirectory"`) do set OPENSEARCH_TMPDIR=%%a
+ for /f "tokens=* usebackq" %%a in (`CALL "%JAVA%" -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.TempDirectory"`) do set OPENSEARCH_TMPDIR=%%a
 )
 
 rem The JVM options parser produces the final JVM options to start
@@ -88,7 +88,7 @@ rem jvm.options.d/*.options
 rem - third, JVM options from OPENSEARCH_JAVA_OPTS are applied
 rem - fourth, ergonomic JVM options are applied
 @setlocal
-for /F "usebackq delims=" %%a in (`CALL %JAVA% -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.JvmOptionsParser" "!OPENSEARCH_PATH_CONF!" ^|^| echo jvm_options_parser_failed`) do set OPENSEARCH_JAVA_OPTS=%%a
+for /F "usebackq delims=" %%a in (`CALL "%JAVA%" -cp "!OPENSEARCH_CLASSPATH!" "org.opensearch.tools.launchers.JvmOptionsParser" "!OPENSEARCH_PATH_CONF!" ^|^| echo jvm_options_parser_failed`) do set OPENSEARCH_JAVA_OPTS=%%a
 @endlocal & set "MAYBE_JVM_OPTIONS_PARSER_FAILED=%OPENSEARCH_JAVA_OPTS%" & set OPENSEARCH_JAVA_OPTS=%OPENSEARCH_JAVA_OPTS%
 
 if "%MAYBE_JVM_OPTIONS_PARSER_FAILED%" == "jvm_options_parser_failed" (
@@ -103,7 +103,7 @@ SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^<=^^^<!
 SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^>=^^^>!
 SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^\=^^^\!
 
-ECHO.!KEYSTORE_PASSWORD!| %JAVA% %OPENSEARCH_JAVA_OPTS% -Dopensearch ^
+ECHO.!KEYSTORE_PASSWORD!| "%JAVA%" %OPENSEARCH_JAVA_OPTS% -Dopensearch ^
  -Dopensearch.path.home="%OPENSEARCH_HOME%" -Dopensearch.path.conf="%OPENSEARCH_PATH_CONF%" ^
  -Dopensearch.distribution.type="%OPENSEARCH_DISTRIBUTION_TYPE%" ^
  -Dopensearch.bundled_jdk="%OPENSEARCH_BUNDLED_JDK%" ^

distribution/tools/upgrade-cli/licenses/jackson-annotations-2.13.4.jar.sha1

@@ -0,0 +1 @@
+858c6cc78e1f08a885b1613e1d817c829df70a6e

distribution/tools/upgrade-cli/licenses/jackson-databind-2.13.4.jar.sha1

@@ -0,0 +1 @@
+98b0edfa8e4084078f10b7b356c300ded4a71491

libs/x-content/licenses/jackson-core-2.13.4.jar.sha1

@@ -0,0 +1 @@
+0cf934c681294b97ef6d80082faeefbe1edadf56

libs/x-content/licenses/jackson-dataformat-cbor-2.13.4.jar.sha1

@@ -0,0 +1 @@
+ccaf21e6a02a20cae6591a12d20bf310544cf3ee

libs/x-content/licenses/jackson-dataformat-smile-2.13.4.jar.sha1

@@ -0,0 +1 @@
+4161a7c3914a12e7b7940ea53eb3c53e17aea91b

libs/x-content/licenses/jackson-dataformat-yaml-2.13.4.jar.sha1

@@ -0,0 +1 @@
+3142ec201e878372d1561e64bd1a947d9e88a03d

libs/x-content/licenses/snakeyaml-1.32.jar.sha1

@@ -0,0 +1 @@
+e80612549feb5c9191c498de628c1aa80693cf0b

modules/ingest-geoip/licenses/jackson-annotations-2.13.4.jar.sha1

@@ -0,0 +1 @@
+858c6cc78e1f08a885b1613e1d817c829df70a6e

modules/ingest-geoip/licenses/jackson-databind-2.13.4.jar.sha1

@@ -0,0 +1 @@
+98b0edfa8e4084078f10b7b356c300ded4a71491

modules/transport-netty4/build.gradle

@@ -143,14 +143,6 @@ thirdPartyAudit {
  'org.apache.log4j.Level',
  'org.apache.log4j.Logger',
 
- // from io.netty.handler.ssl.OpenSslEngine (netty)
- 'org.bouncycastle.openssl.PEMEncryptedKeyPair',
- 'org.bouncycastle.openssl.PEMParser',
- 'org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter',
- 'org.bouncycastle.openssl.jcajce.JceOpenSSLPKCS8DecryptorProviderBuilder',
- 'org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder',
- 'org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo',
-
  // from io.netty.handler.ssl.OpenSslEngine (netty)
  'io.netty.internal.tcnative.Buffer',
  'io.netty.internal.tcnative.CertificateCompressionAlgo',

modules/transport-netty4/licenses/netty-buffer-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+6c014412b599489b1db27c6bc08d8a46da94e397

modules/transport-netty4/licenses/netty-codec-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+18f5b02af7ca611978bc28f2cb58cbb3b9b0f0ef

modules/transport-netty4/licenses/netty-codec-http-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+882c70bc0a30a98bf3ce477f043e967ac026044c

modules/transport-netty4/licenses/netty-common-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+2814bd465731355323aba0fdd22163bfce638a75

modules/transport-netty4/licenses/netty-handler-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+2dc22423c8ed19906615fb936a5fcb7db14a4e6c

modules/transport-netty4/licenses/netty-resolver-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+55ecb1ff4464b56564a90824a741c3911264aaa4

modules/transport-netty4/licenses/netty-transport-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+6cc2b49749b4fbcc39c687027e04e65e857552a9

modules/transport-netty4/licenses/netty-transport-native-unix-common-4.1.79.Final.jar.sha1

@@ -0,0 +1 @@
+731937caec938b77b39df932a8da8aaca8d5ec05

plugins/discovery-ec2/licenses/jackson-annotations-2.13.4.jar.sha1

@@ -0,0 +1 @@
+858c6cc78e1f08a885b1613e1d817c829df70a6e

plugins/discovery-ec2/licenses/jackson-databind-2.13.4.jar.sha1

@@ -0,0 +1 @@
+98b0edfa8e4084078f10b7b356c300ded4a71491

plugins/repository-azure/licenses/jackson-annotations-2.13.4.jar.sha1

@@ -0,0 +1 @@
+858c6cc78e1f08a885b1613e1d817c829df70a6e

plugins/repository-azure/licenses/jackson-databind-2.13.4.jar.sha1

@@ -0,0 +1 @@
+98b0edfa8e4084078f10b7b356c300ded4a71491