Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into 7363
Browse files Browse the repository at this point in the history
Signed-off-by: Ashish Singh <ssashish@amazon.com>
  • Loading branch information
ashking94 committed May 4, 2023
2 parents ea84f4f + 01e5c29 commit 27c2312
Show file tree
Hide file tree
Showing 104 changed files with 3,252 additions and 3,527 deletions.
1 change: 1 addition & 0 deletions .ci/bwcVersions
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,5 @@ BWC_VERSION:
- "2.6.0"
- "2.6.1"
- "2.7.0"
- "2.7.1"
- "2.8.0"
3 changes: 3 additions & 0 deletions .idea/inspectionProfiles/Project_Default.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

23 changes: 15 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `wiremock-jre8-standalone` from 2.33.2 to 2.35.0
- Bump `gson` from 2.10 to 2.10.1
- Bump `json-schema-validator` from 1.0.73 to 1.0.76
- Bump `jna` from 5.11.0 to 5.13.0
- Bump `joni` from 2.1.44 to 2.1.45
- Bump `commons-io:commons-io` from 2.7 to 2.11.0
- Bump `org.jruby.joni:joni` from 2.1.45 to 2.1.48
- Bump `com.google.code.gson:gson` from 2.10 to 2.10.1
- Bump `com.maxmind.geoip2:geoip2` from 4.0.0 to 4.0.1
- Bump `com.networknt:json-schema-validator` from 1.0.76 to 1.0.78
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.0.0 to 12.1.0
- Bump `com.avast.gradle:gradle-docker-compose-plugin` from 0.16.11 to 0.16.12
- Bump `org.apache.commons:commons-compress` from 1.22 to 1.23.0
- Bump `org.apache.commons:commons-configuration2` from 2.8.0 to 2.9.0
- Bump `com.netflix.nebula:nebula-publishing-plugin` from 19.2.0 to 20.3.0
- Bump `com.netflix.nebula.ospackage-base` from 11.0.0 to 11.3.0
- Bump `org.apache.commons:commons-compress` from 1.22 to 1.23.0
- Bump `com.google.protobuf:protobuf-java` from 3.22.2 to 3.22.3
- Bump `com.diffplug.spotless` from 6.17.0 to 6.18.0
- Bump `io.opencensus:opencensus-api` from 0.18.0 to 0.31.1 ([#7291](https://github.com/opensearch-project/OpenSearch/pull/7291))

Expand Down Expand Up @@ -84,14 +79,26 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

## [Unreleased 2.x]
### Added
- [Extensions] Moving Extensions APIs to protobuf serialization. ([#6960](https://github.com/opensearch-project/OpenSearch/pull/6960))
- [Extensions] Moving Extensions APIs to support cross versions via protobuf. ([#7402](https://github.com/opensearch-project/OpenSearch/issues/7402))
- [Extensions] Add IdentityPlugin into core to support Extension identities ([#7246](https://github.com/opensearch-project/OpenSearch/pull/7246))
- Add connectToNodeAsExtension in TransportService ([#6866](https://github.com/opensearch-project/OpenSearch/pull/6866))
- Add descending order search optimization through reverse segment read. ([#7244](https://github.com/opensearch-project/OpenSearch/pull/7244))

### Dependencies
- Bump `jackson` from 2.14.2 to 2.15.0 ([#7286](https://github.com/opensearch-project/OpenSearch/pull/7286)
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.0.0 to 12.1.0
- Bump `com.netflix.nebula:nebula-publishing-plugin` from 19.2.0 to 20.2.0
- Bump `com.google.protobuf:protobuf-java` from 3.22.2 to 3.22.3
- Bump `jackson` from 2.14.2 to 2.15.0 ([#7286](https://github.com/opensearch-project/OpenSearch/pull/7286))
- Bump `com.netflix.nebula:nebula-publishing-plugin` from 20.2.0 to 20.3.0
- Bump `com.netflix.nebula.ospackage-base` from 11.0.0 to 11.3.0
- Bump `gradle.plugin.com.github.johnrengelman:shadow` from 7.1.2 to 8.0.0
- Bump `jna` from 5.11.0 to 5.13.0
- Bump `commons-io:commons-io` from 2.7 to 2.11.0
- Bump `org.apache.shiro:shiro-core` from 1.9.1 to 1.11.0 ([#7397](https://github.com/opensearch-project/OpenSearch/pull/7397)

### Changed
- Enable `./gradlew build` on MacOS by disabling bcw tests ([#7303](https://github.com/opensearch-project/OpenSearch/pull/7303))
- Moved concurrent-search from sandbox plugin to server module behind feature flag ([#7203](https://github.com/opensearch-project/OpenSearch/pull/7203))

### Deprecated

Expand All @@ -102,4 +109,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.5...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.7...2.x
36 changes: 33 additions & 3 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
- [Windows](#windows)
- [Docker](#docker)
- [Build](#build)
- [Generated Code](#generated-code)
- [Run Tests](#run-tests)
- [Run OpenSearch](#run-opensearch)
- [Use an Editor](#use-an-editor)
Expand All @@ -26,6 +27,7 @@
- [`server`](#server)
- [`test`](#test)
- [Java Language Formatting Guidelines](#java-language-formatting-guidelines)
- [Adding Dependencies](#adding-dependencies)
- [Editor / IDE Support](#editor--ide-support)
- [Formatting Failures](#formatting-failures)
- [Gradle Build](#gradle-build)
Expand All @@ -39,7 +41,7 @@
- [Distribution Download Plugin](#distribution-download-plugin)
- [Creating fat-JAR of a Module](#creating-fat-jar-of-a-module)
- [Components](#components)
- [Build libraries & interfaces](#build-libraries--interfaces)
- [Build Libraries & Interfaces](#build-libraries--interfaces)
- [Clients & Libraries](#clients--libraries)
- [Plugins](#plugins-1)
- [Indexing & Search](#indexing--search)
Expand Down Expand Up @@ -133,6 +135,17 @@ To build a distribution to run on your local platform, run:

All distributions built will be under `distributions/archives`.

#### Generated Code

OpenSearch uses code generators like [Protobuf](https://protobuf.dev/).
OpenSearch build system already takes a dependency of generating code from protobuf, incase you run into compilation errors, run:

```
./gradlew generateProto
```

Generated code in OpenSearch is used to establish cross version compatibility communication for API contracts within OpenSearch.

### Run Tests

OpenSearch uses a Gradle wrapper for its build. Run `gradlew` on Unix systems, or `gradlew.bat` on Windows in the root of the repository.
Expand Down Expand Up @@ -328,6 +341,18 @@ Please follow these formatting guidelines:
* Note that JavaDoc and block comments i.e. `/* ... */` are not formatted, but line comments i.e `// ...` are.
* There is an implicit rule that negative boolean expressions should use the form `foo == false` instead of `!foo` for better readability of the code. While this isn't strictly enforced, if might get called out in PR reviews as something to change.

## Adding Dependencies

When adding a new dependency or removing an existing dependency via any `build.gradle` (that are not in the test scope), update the dependency LICENSE and library SHAs.

For example, after adding `api "org.slf4j:slf4j-api:${versions.slf4j}"` to [plugins/discovery-ec2/build.gradle](plugins/discovery-ec2/build.gradle), copy the library `LICENSE.txt` and `NOTICE.txt` to `plugins/discovery-ec2/licenses/slf4j-api-LICENSE.txt` and `plugins/discovery-ec2/licenses/slf4j-api-NOTICE.txt`, then run the following to generate `plugins/discovery-ec2/licenses/slf4j-api-1.7.36.jar.sha1`.

```
./gradlew :plugins:discovery-ec2:updateSHAs
```

Ensure that `./gradlew :plugins:discovery-ec2:check` passes before submitting changes.

### Editor / IDE Support

IntelliJ IDEs can [import](https://blog.jetbrains.com/idea/2014/01/intellij-idea-13-importing-code-formatter-settings-from-eclipse/) the [settings file](buildSrc/formatterConfig.xml), and / or use the [Eclipse Code Formatter](https://plugins.jetbrains.com/plugin/6546-eclipse-code-formatter)
Expand Down Expand Up @@ -424,7 +449,7 @@ Refer the installed JAR as any other maven artifact, e.g.

As you work in the OpenSearch repo you may notice issues getting labeled with component labels. It's a housekeeping task to help group together similar pieces of work. You can pretty much ignore it, but if you're curious, here's what the different labels mean:

### Build libraries & interfaces
### Build Libraries & Interfaces

Tasks to make sure the build tasks are useful and packaging and distribution are easy.

Expand All @@ -437,7 +462,6 @@ Includes:
- Compatibility
- Javadoc enforcement


### Clients & Libraries

APIs and communication mechanisms for external connections to OpenSearch. This includes the “library” directory in OpenSearch (a set of common functions).
Expand Down Expand Up @@ -499,6 +523,7 @@ Includes:
Security is our top priority. Avoid checking in credentials.

#### Installation

Install [awslabs/git-secrets](https://github.com/awslabs/git-secrets) by running the following commands.
```
git clone https://github.com/awslabs/git-secrets.git
Expand All @@ -507,6 +532,7 @@ make install
```

#### Configuration

You can configure git secrets per repository, you need to change the directory to the root of the repository and run the following command.
```
git secrets --install
Expand All @@ -533,11 +559,13 @@ OpenSearch addresses backward and forward compatibility at three different level
to ensure backwards compatibility are provided below.

#### Data

The data level consists of index and application data file formats. OpenSearch guarantees file formats and indexes are compatible only back to the first release of
the previous major version. If on disk formats or encodings need to be changed (including index data, cluster state, or any other persisted data) developers must
use Version checks accordingly (e.g., `Version.onOrAfter`, `Version.before`) to guarantee backwards compatibility.

#### Developer API

The Developer API consists of interfaces and foundation software implementations that enable external users to develop new OpenSearch features. This includes
obvious components such as the Plugin framework and less obvious components such as REST Action Handlers. When developing a new feature of OpenSearch it is important
to explicitly mark which implementation components may, or may not, be extended by external implementations. For example, all new API classes with `@opensearch.api`
Expand All @@ -547,6 +575,7 @@ guarantee backwards compatibility and may change at any time. The `@deprecated`
either changed or planned to be removed across minor versions.

#### User API

The User API consists of integration specifications (e.g., [Query Domain Specific Language](https://opensearch.org/docs/latest/opensearch/query-dsl/index/),
[field mappings](https://opensearch.org/docs/latest/opensearch/mappings/)) and endpoints (e.g., [`_search`](https://opensearch.org/docs/latest/api-reference/search/),
[`_cat`](https://opensearch.org/docs/latest/api-reference/cat/index/)) users rely on to integrate and use OpenSearch. Backwards compatibility is critical to the
Expand All @@ -556,6 +585,7 @@ users of any changes by adding the `>breaking` label on Pull Requests, adding an
and a log message to the OpenSearch deprecation log files using the `DeprecationLogger`.

#### Experimental Development

Rapidly developing new features often benefit from several release cycles before committing to an official and long term supported (LTS) API. To enable this cycle OpenSearch
uses an Experimental Development process leveraging [Feature Flags](https://featureflags.io/feature-flags/). This allows a feature to be developed using the same process as
a LTS feature but with additional guard rails and communication mechanisms to signal to the users and development community the feature is not yet stable, may change in a future
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
[![Security Vulnerabilities](https://img.shields.io/github/issues/opensearch-project/OpenSearch/security%20vulnerability?labelColor=red)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"security%20vulnerability")
[![Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch)](https://github.com/opensearch-project/OpenSearch/issues)
[![Open Pull Requests](https://img.shields.io/github/issues-pr/opensearch-project/OpenSearch)](https://github.com/opensearch-project/OpenSearch/pulls)
[![2.6 Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch/v2.6.0)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"v2.6.0")
[![2.8 Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch/v2.8.0)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"v2.8.0")
[![3.0 Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch/v3.0.0)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"v3.0.0")
[![GHA gradle check](https://github.com/opensearch-project/OpenSearch/actions/workflows/gradle-check.yml/badge.svg)](https://github.com/opensearch-project/OpenSearch/actions/workflows/gradle-check.yml)
[![GHA validate pull request](https://github.com/opensearch-project/OpenSearch/actions/workflows/wrapper.yml/badge.svg)](https://github.com/opensearch-project/OpenSearch/actions/workflows/wrapper.yml)
Expand Down
2 changes: 1 addition & 1 deletion buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ dependencies {
api 'org.apache.rat:apache-rat:0.15'
api 'commons-io:commons-io:2.11.0'
api "net.java.dev.jna:jna:5.13.0"
api 'gradle.plugin.com.github.johnrengelman:shadow:7.1.2'
api 'gradle.plugin.com.github.johnrengelman:shadow:8.0.0'
api 'org.jdom:jdom2:2.0.6.1'
api "org.jetbrains.kotlin:kotlin-stdlib-jdk8:${props.getProperty('kotlin')}"
api 'de.thetaphi:forbiddenapis:3.5.1'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.gradle.testclusters;

public class ExtensionsProperties {
private String name;
private String uniqueId;
private String hostAddress;
private String port;
private String version;
private String opensearchVersion;
private String minimumCompatibleVersion;

public ExtensionsProperties(
String name,
String uniqueId,
String hostAddress,
String port,
String version,
String opensearchVersion,
String minimumCompatibleVersion
) {
this.name = name;
this.uniqueId = uniqueId;
this.hostAddress = hostAddress;
this.port = port;
this.version = version;
this.opensearchVersion = opensearchVersion;
this.minimumCompatibleVersion = minimumCompatibleVersion;
}

public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

public String getUniqueId() {
return uniqueId;
}

public void setUniqueId(String uniqueId) {
this.uniqueId = uniqueId;
}

public String getHostAddress() {
return hostAddress;
}

public void setHostAddress(String hostAddress) {
this.hostAddress = hostAddress;
}

public String getPort() {
return port;
}

public void setPort(String port) {
this.port = port;
}

public String getVersion() {
return version;
}

public void setVersion(String version) {
this.version = version;
}

public String getOpensearchVersion() {
return opensearchVersion;
}

public void setOpensearchVersion(String opensearchVersion) {
this.opensearchVersion = opensearchVersion;
}

public String getMinimumCompatibleVersion() {
return minimumCompatibleVersion;
}

public void setMinimumCompatibleVersion(String minimumCompatibleVersion) {
this.minimumCompatibleVersion = minimumCompatibleVersion;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,11 @@ public void setTestDistribution(TestDistribution distribution) {
nodes.all(each -> each.setTestDistribution(distribution));
}

@Override
public void extension(ExtensionsProperties extension) {
nodes.all(each -> each.extension(extension));
}

@Override
public void plugin(Provider<RegularFile> plugin) {
nodes.all(each -> each.plugin(plugin));
Expand Down
Loading

0 comments on commit 27c2312

Please sign in to comment.