Skip to content

Commit

Permalink
Merge branch 'main' into restrict-remote-store-index-settings
Browse files Browse the repository at this point in the history
Signed-off-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com>
  • Loading branch information
gbbafna authored Aug 1, 2023
2 parents 839aa94 + 77074b4 commit 194f610
Show file tree
Hide file tree
Showing 632 changed files with 4,923 additions and 2,063 deletions.
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Added
- Add server version as REST response header [#6583](https://github.com/opensearch-project/OpenSearch/issues/6583)
- Start replication checkpointTimers on primary before segments upload to remote store. ([#8221]()https://github.com/opensearch-project/OpenSearch/pull/8221)
- [distribution/archives] [Linux] [x64] Provide the variant of the distributions bundled with JRE ([#8195]()https://github.com/opensearch-project/OpenSearch/pull/8195)
- Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache ([#8606](https://github.com/opensearch-project/OpenSearch/pull/8606))

### Dependencies
- Bump `org.apache.logging.log4j:log4j-core` from 2.17.1 to 2.20.0 ([#8307](https://github.com/opensearch-project/OpenSearch/pull/8307))
Expand All @@ -91,6 +93,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `com.google.http-client:google-http-client-gson` from 1.43.2 to 1.43.3 ([#8840](https://github.com/opensearch-project/OpenSearch/pull/8840))
- OpenJDK Update (July 2023 Patch releases) ([#8868](https://github.com/opensearch-project/OpenSearch/pull/8868)
- Bump `hadoop` libraries from 3.3.4 to 3.3.6 ([#6995](https://github.com/opensearch-project/OpenSearch/pull/6995))
- Bump `com.gradle.enterprise` from 3.13.3 to 3.14.1 ([#8996](https://github.com/opensearch-project/OpenSearch/pull/8996))
- Bump `org.apache.commons:commons-lang3` from 3.12.0 to 3.13.0 ([#8995](https://github.com/opensearch-project/OpenSearch/pull/8995))
- Bump `com.google.cloud:google-cloud-core-http` from 2.21.0 to 2.21.1 ([#8999](https://github.com/opensearch-project/OpenSearch/pull/8999))

### Changed
- Perform aggregation postCollection in ContextIndexSearcher after searching leaves ([#8303](https://github.com/opensearch-project/OpenSearch/pull/8303))
Expand All @@ -99,12 +104,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Exclude 'benchmarks' from codecov report ([#8805](https://github.com/opensearch-project/OpenSearch/pull/8805))
- [Refactor] MediaTypeParser to MediaTypeParserRegistry ([#8636](https://github.com/opensearch-project/OpenSearch/pull/8636))
- Create separate SourceLookup instance per segment slice in SignificantTextAggregatorFactory ([#8807](https://github.com/opensearch-project/OpenSearch/pull/8807))
- Add support for aggregation profiler with concurrent aggregation ([#8801](https://github.com/opensearch-project/OpenSearch/pull/8801))
- [Remove] Deprecated Fractional ByteSizeValue support #9005 ([#9005](https://github.com/opensearch-project/OpenSearch/pull/9005))

### Deprecated

### Removed

### Fixed
- Fix flaky ResourceAwareTasksTests.testBasicTaskResourceTracking test ([#8993](https://github.com/opensearch-project/OpenSearch/pull/8993))

### Security

Expand Down
5 changes: 4 additions & 1 deletion DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,10 @@ This repository is split into many top level directories. The most important one

### `distribution`

Builds our tar and zip archives and our rpm and deb packages.
Builds our tar and zip archives and our rpm and deb packages. There are several flavors of the distributions, with the classifier included in the name of the final deliverable (archive or package):
- default (no classifier), the distribution with bundled JDK
- `-no-jdk-` - the distribution without bundled JDK/JRE, assumes the JDK/JRE is going to be pre-installed on the target systems
- `-jre-` - the distribution bundled with JRE (smaller footprint), supported as experimental feature for some platforms

### `libs`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,14 @@
import org.opensearch.action.search.SearchPhaseController;
import org.opensearch.action.search.SearchProgressListener;
import org.opensearch.action.search.SearchRequest;
import org.opensearch.common.breaker.CircuitBreaker;
import org.opensearch.common.breaker.NoopCircuitBreaker;
import org.opensearch.core.common.breaker.CircuitBreaker;
import org.opensearch.core.common.breaker.NoopCircuitBreaker;
import org.opensearch.core.common.io.stream.NamedWriteableRegistry;
import org.opensearch.common.lucene.search.TopDocsAndMaxScore;
import org.opensearch.common.settings.Settings;
import org.opensearch.core.index.Index;
import org.opensearch.core.index.shard.ShardId;
import org.opensearch.indices.breaker.NoneCircuitBreakerService;
import org.opensearch.core.indices.breaker.NoneCircuitBreakerService;
import org.opensearch.search.DocValueFormat;
import org.opensearch.search.SearchModule;
import org.opensearch.search.SearchShardTarget;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@
import org.openjdk.jmh.annotations.Threads;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.infra.Blackhole;
import org.opensearch.common.breaker.CircuitBreaker;
import org.opensearch.common.breaker.NoopCircuitBreaker;
import org.opensearch.core.common.breaker.CircuitBreaker;
import org.opensearch.core.common.breaker.NoopCircuitBreaker;
import org.opensearch.index.store.remote.filecache.CachedIndexInput;
import org.opensearch.index.store.remote.filecache.FileCache;
import org.opensearch.index.store.remote.filecache.FileCacheFactory;
Expand Down
1 change: 0 additions & 1 deletion build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,6 @@ subprojects {
includeClasses.add("org.opensearch.cluster.metadata.IndexGraveyardTests")
includeClasses.add("org.opensearch.cluster.routing.MovePrimaryFirstTests")
includeClasses.add("org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT")
includeClasses.add("org.opensearch.cluster.service.MasterServiceTests")
includeClasses.add("org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests")
includeClasses.add("org.opensearch.gateway.RecoveryFromGatewayIT")
includeClasses.add("org.opensearch.gateway.ReplicaShardAllocatorIT")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,10 @@
* compatible open source license.
*/

/** Base Lifecycle Component package. */
package org.opensearch.common.component;
package org.opensearch.gradle;

public enum JavaPackageType {
NONE,
JRE,
JDK
}
197 changes: 197 additions & 0 deletions buildSrc/src/main/java/org/opensearch/gradle/JavaVariant.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.gradle;

import org.gradle.api.Buildable;
import org.gradle.api.artifacts.Configuration;
import org.gradle.api.model.ObjectFactory;
import org.gradle.api.provider.Property;
import org.gradle.api.tasks.TaskDependency;

import java.io.File;
import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

abstract class JavaVariant implements Buildable, Iterable<File> {

private static final List<String> ALLOWED_ARCHITECTURES = Collections.unmodifiableList(
Arrays.asList("aarch64", "x64", "s390x", "ppc64le")
);
private static final List<String> ALLOWED_VENDORS = Collections.unmodifiableList(Arrays.asList("adoptium", "adoptopenjdk", "openjdk"));
private static final List<String> ALLOWED_PLATFORMS = Collections.unmodifiableList(
Arrays.asList("darwin", "freebsd", "linux", "mac", "windows")
);
private static final Pattern VERSION_PATTERN = Pattern.compile("(\\d+)(\\.\\d+\\.\\d+)?\\+(\\d+(?:\\.\\d+)?)(@([a-f0-9]{32}))?");
private static final Pattern LEGACY_VERSION_PATTERN = Pattern.compile("(\\d)(u\\d+)(?:\\+|\\-)(b\\d+?)(@([a-f0-9]{32}))?");

private final String name;
private final Configuration configuration;

private final Property<String> vendor;
private final Property<String> version;
private final Property<String> platform;
private final Property<String> architecture;
private String baseVersion;
private String major;
private String build;
private String hash;

JavaVariant(String name, Configuration configuration, ObjectFactory objectFactory) {
this.name = name;
this.configuration = configuration;
this.vendor = objectFactory.property(String.class);
this.version = objectFactory.property(String.class);
this.platform = objectFactory.property(String.class);
this.architecture = objectFactory.property(String.class);
}

public String getName() {
return name;
}

public String getVendor() {
return vendor.get();
}

public void setVendor(final String vendor) {
if (ALLOWED_VENDORS.contains(vendor) == false) {
throw new IllegalArgumentException("unknown vendor [" + vendor + "] for jdk [" + name + "], must be one of " + ALLOWED_VENDORS);
}
this.vendor.set(vendor);
}

public String getVersion() {
return version.get();
}

public void setVersion(String version) {
if (VERSION_PATTERN.matcher(version).matches() == false && LEGACY_VERSION_PATTERN.matcher(version).matches() == false) {
throw new IllegalArgumentException("malformed version [" + version + "] for jdk [" + name + "]");
}
parseVersion(version);
this.version.set(version);
}

public String getPlatform() {
return platform.get();
}

public void setPlatform(String platform) {
if (ALLOWED_PLATFORMS.contains(platform) == false) {
throw new IllegalArgumentException(
"unknown platform [" + platform + "] for jdk [" + name + "], must be one of " + ALLOWED_PLATFORMS
);
}
this.platform.set(platform);
}

public String getArchitecture() {
return architecture.get();
}

public void setArchitecture(final String architecture) {
String jdkArchitecture = translateJdkArchitecture(architecture);
if (ALLOWED_ARCHITECTURES.contains(jdkArchitecture) == false) {
throw new IllegalArgumentException(
"unknown architecture [" + jdkArchitecture + "] for jdk [" + name + "], must be one of " + ALLOWED_ARCHITECTURES
);
}
this.architecture.set(jdkArchitecture);
}

public String getBaseVersion() {
return baseVersion;
}

public String getMajor() {
return major;
}

public String getBuild() {
return build;
}

public String getHash() {
return hash;
}

public String getPath() {
return configuration.getSingleFile().toString();
}

public String getConfigurationName() {
return configuration.getName();
}

@Override
public String toString() {
return getPath();
}

@Override
public TaskDependency getBuildDependencies() {
return configuration.getBuildDependencies();
}

// internal, make this jdks configuration unmodifiable
void finalizeValues() {
if (version.isPresent() == false) {
throw new IllegalArgumentException("version not specified for jdk [" + name + "]");
}
if (platform.isPresent() == false) {
throw new IllegalArgumentException("platform not specified for jdk [" + name + "]");
}
if (vendor.isPresent() == false) {
throw new IllegalArgumentException("vendor not specified for jdk [" + name + "]");
}
if (architecture.isPresent() == false) {
throw new IllegalArgumentException("architecture not specified for jdk [" + name + "]");
}
version.finalizeValue();
platform.finalizeValue();
vendor.finalizeValue();
architecture.finalizeValue();
}

@Override
public Iterator<File> iterator() {
return configuration.iterator();
}

private void parseVersion(String version) {
// decompose the bundled jdk version, broken into elements as: [feature, interim, update, build]
// Note the "patch" version is not yet handled here, as it has not yet been used by java.
Matcher jdkVersionMatcher = VERSION_PATTERN.matcher(version);
if (jdkVersionMatcher.matches() == false) {
// Try again with the pre-Java9 version format
jdkVersionMatcher = LEGACY_VERSION_PATTERN.matcher(version);

if (jdkVersionMatcher.matches() == false) {
throw new IllegalArgumentException("Malformed jdk version [" + version + "]");
}
}

baseVersion = jdkVersionMatcher.group(1) + (jdkVersionMatcher.group(2) != null ? (jdkVersionMatcher.group(2)) : "");
major = jdkVersionMatcher.group(1);
build = jdkVersionMatcher.group(3);
hash = jdkVersionMatcher.group(5);
}

private String translateJdkArchitecture(String architecture) {
/*
* Jdk uses aarch64 from ARM. Translating from arm64 to aarch64 which Jdk understands.
*/
return "arm64".equals(architecture) ? "aarch64" : architecture;
}

}
Loading

0 comments on commit 194f610

Please sign in to comment.