Merge branch '6.x' into ccr-6.x

* 6.x:
  Fix scriptdocvalues tests with dates
  Correct minor typo in explain.asciidoc for HLRC
  Fix painless whitelist and warnings from backporting #31441
  Build: Add elastic maven to repos used by BuildPlugin (#32549)
  Scripting: Conditionally use java time api in scripting (#31441)
  [ML] Improve error when no available field exists for rule scope (#32550)
  [ML] Improve error for functions with limited rule condition support (#32548)
  [ML] Remove multiple_bucket_spans
  [ML] Fix thread leak when waiting for job flush (#32196) (#32541)
  Painless: Clean Up PainlessField (#32525)
  Add @AwaitsFix for #32554
  Remove broken @link in Javadoc
  Add AwaitsFix to failing test - see #32546
  SQL: Added support for string manipulating functions with more than one parameter (#32356)
  [DOCS] Reloadable Secure Settings (#31713)
  Fix compilation error introduced by #32339
  [Rollup] Remove builders from TermsGroupConfig (#32507)
  Use hostname instead of IP with SPNEGO test (#32514)
  Switch x-pack rolling restart to new style Requests (#32339)
  [DOCS] Small fixes in rule configuration page (#32516)
  Painless: Clean up PainlessMethod (#32476)
  SQL: Add test for handling of partial results (#32474)
  Docs: Add missing migration doc for logging change
  Build: Remove shadowing from benchmarks (#32475)
  Docs: Add all JDKs to CONTRIBUTING.md
  Logging: Make node name consistent in logger (#31588)
  High-level client: fix clusterAlias parsing in SearchHit (#32465)
  REST high-level client: parse back _ignored meta field (#32362)
  backport fix of reduceRandom fix (#32508)
  Add licensing enforcement for FIPS mode (#32437)
  INGEST: Clean up Java8 Stream Usage (#32059) (#32485)
  Improve the error message when an index is incompatible with field aliases. (#32482)
  Mute testFilterCacheStats
  Scripting: Fix painless compiler loader to know about context classes (#32385)
  [ML][DOCS] Fix typo applied_to => applies_to
  Mute SSLTrustRestrictionsTests on JDK 11
  Changed ReindexRequest to use Writeable.Reader (#32401)
  Increase max chunk size to 256Mb for repo-azure (#32101)
  Mute KerberosAuthenticationIT
  fix no=>not typo (#32463)
  HLRC: Add delete watch action (#32337)
  Fix calculation of orientation of polygons (#27967)
  [Kerberos] Add missing javadocs (#32469)
  Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
  Make get all app privs requires "*" permission (#32460)
  Ensure KeyStoreWrapper decryption exceptions are handled (#32472)
  update rollover to leverage write-alias semantics (#32216)
  [Kerberos] Remove Kerberos bootstrap checks (#32451)
  Switch security to new style Requests (#32290)
  Switch security spi example to new style Requests (#32341)
  Painless: Add PainlessConstructor (#32447)
  Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819)
  Remove > from Javadoc (fatal with Java 11)
  Tests: Fix convert error tests to use fixed value (#32415)
  IndicesClusterStateService should replace an init. replica with an init. primary with the same aId (#32374)
  auto-interval date histogram - 6.x backport (#32107)
  [CI] Mute DocumentSubsetReaderTests testSearch
  [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
  TEST: testDocStats should always use forceMerge (#32450)
  TEST: Avoid deletion in FlushIT
  AwaitsFix IndexShardTests#testDocStats
  Painless: Add method type to method. (#32441)
  Remove reference to non-existent store type (#32418)
  [TEST] Mute failing FlushIT test
  Fix ordering of bootstrap checks in docs (#32417)
  Wrong discovery.type for azure in breaking changes (#32432)
  Mute ConvertProcessorTests failing tests
  TESTS: Move netty leak detection to paranoid level (#32354) (#32425)
  Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
  [Kerberos] Avoid vagrant update on precommit (#32416)
  TEST: Avoid triggering merges in FlushIT
  [DOCS] Fixes formatting of scope object in job resource
  Switch x-pack/plugin to new style Requests (#32327)
  Release requests in cors handle (#32410)
  Remove BouncyCastle dependency from runtime (#32402)
  Copy missing segment attributes in getSegmentInfo (#32396)
  Rest HL client: Add put license action (#32214)
  Docs: Correcting a typo in tophits (#32359)
  Build: Stop double generating buildSrc pom (#32408)
  Switch x-pack full restart to new style Requests (#32294)
  Painless: Clean Up PainlessClass Variables (#32380)
  [ML] Consistent pattern for strict/lenient parser names (#32399)
  Add Restore Snapshot High Level REST API
  Update update-settings.asciidoc (#31378)
  Introduce index store plugins (#32375)
  Rank-Eval: Reduce scope of an unchecked supression
  Make sure _forcemerge respects `max_num_segments`. (#32291)

CONTRIBUTING.md

@@ -100,6 +100,12 @@ JDK 10 and testing on a JDK 8 runtime; to do this, set `RUNTIME_JAVA_HOME`
 pointing to the Java home of a JDK 8 installation. Note that this mechanism can
 be used to test against other JDKs as well, this is not only limited to JDK 8.
 
+> Note: It is also required to have `JAVA7_HOME`, `JAVA8_HOME` and
+`JAVA10_HOME` available so that the tests can pass.
+
+> Warning: do not use `sdkman` for Java installations which do not have proper
+`jrunscript` for jdk distributions.
+
 Elasticsearch uses the Gradle wrapper for its build. You can execute Gradle
 using the wrapper via the `gradlew` script in the root of the repository.
 

benchmarks/README.md

@@ -4,36 +4,39 @@ This directory contains the microbenchmark suite of Elasticsearch. It relies on
 
 ## Purpose
 
-We do not want to microbenchmark everything but the kitchen sink and should typically rely on our 
-[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/app/kibana#/dashboard/Nightly-Benchmark-Overview) with 
-[Rally](http://github.com/elastic/rally). Microbenchmarks are intended to spot performance regressions in performance-critical components. 
+We do not want to microbenchmark everything but the kitchen sink and should typically rely on our
+[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/app/kibana#/dashboard/Nightly-Benchmark-Overview) with
+[Rally](http://github.com/elastic/rally). Microbenchmarks are intended to spot performance regressions in performance-critical components.
 The microbenchmark suite is also handy for ad-hoc microbenchmarks but please remove them again before merging your PR.
 
 ## Getting Started
 
-Just run `gradle :benchmarks:jmh` from the project root directory. It will build all microbenchmarks, execute them and print the result.
+Just run `gradlew -p benchmarks run` from the project root
+directory. It will build all microbenchmarks, execute them and print
+the result.
 
 ## Running Microbenchmarks
 
-Benchmarks are always run via Gradle with `gradle :benchmarks:jmh`.
-
-Running via an IDE is not supported as the results are meaningless (we have no control over the JVM running the benchmarks).
+Running via an IDE is not supported as the results are meaningless
+because we have no control over the JVM running the benchmarks.
 
-If you want to run a specific benchmark class, e.g. `org.elasticsearch.benchmark.MySampleBenchmark` or have special requirements 
-generate the uberjar with `gradle :benchmarks:jmhJar` and run it directly with:
+If you want to run a specific benchmark class like, say,
+`MemoryStatsBenchmark`, you can use `--args`:
 
 ```
-java -jar benchmarks/build/distributions/elasticsearch-benchmarks-*.jar
+gradlew -p benchmarks run --args ' AllocationBenchmark'
 ```
 
-JMH supports lots of command line parameters. Add `-h` to the command above to see the available command line options.
+Everything in the `'` gets sent on the command line to JMH. The leading ` `
+inside the `'`s is important. Without it parameters are sometimes sent to
+gradle.
 
 ## Adding Microbenchmarks
 
-Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the 
+Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the
 [JMH samples](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/).
 
-In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and 
+In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and
 end the class name of a benchmark with `Benchmark`. To have JMH execute a benchmark, annotate the respective methods with `@Benchmark`.
 
 ## Tips and Best Practices
@@ -42,15 +45,15 @@ To get realistic results, you should exercise care when running benchmarks. Here
 
 ### Do
 
-* Ensure that the system executing your microbenchmarks has as little load as possible. Shutdown every process that can cause unnecessary 
+* Ensure that the system executing your microbenchmarks has as little load as possible. Shutdown every process that can cause unnecessary
   runtime jitter. Watch the `Error` column in the benchmark results to see the run-to-run variance.
 * Ensure to run enough warmup iterations to get the benchmark into a stable state. If you are unsure, don't change the defaults.
 * Avoid CPU migrations by pinning your benchmarks to specific CPU cores. On Linux you can use `taskset`.
-* Fix the CPU frequency to avoid Turbo Boost from kicking in and skewing your results. On Linux you can use `cpufreq-set` and the 
+* Fix the CPU frequency to avoid Turbo Boost from kicking in and skewing your results. On Linux you can use `cpufreq-set` and the
   `performance` CPU governor.
 * Vary the problem input size with `@Param`.
 * Use the integrated profilers in JMH to dig deeper if benchmark results to not match your hypotheses:
-    * Run the generated uberjar directly and use `-prof gc` to check whether the garbage collector runs during a microbenchmarks and skews 
+    * Run the generated uberjar directly and use `-prof gc` to check whether the garbage collector runs during a microbenchmarks and skews
    your results. If so, try to force a GC between runs (`-gc true`) but watch out for the caveats.
     * Use `-prof perf` or `-prof perfasm` (both only available on Linux) to see hotspots.
 * Have your benchmarks peer-reviewed.
@@ -59,4 +62,4 @@ To get realistic results, you should exercise care when running benchmarks. Here
 
 * Blindly believe the numbers that your microbenchmark produces but verify them by measuring e.g. with `-prof perfasm`.
 * Run more threads than your number of CPU cores (in case you run multi-threaded microbenchmarks).
-* Look only at the `Score` column and ignore `Error`. Instead take countermeasures to keep `Error` low / variance explainable.
+* Look only at the `Score` column and ignore `Error`. Instead take countermeasures to keep `Error` low / variance explainable.

benchmarks/build.gradle

@@ -18,11 +18,8 @@
  */
 
 apply plugin: 'elasticsearch.build'
-
-// order of this section matters, see: https://github.com/johnrengelman/shadow/issues/336
-apply plugin: 'application' // have the shadow plugin provide the runShadow task
+apply plugin: 'application'
 mainClassName = 'org.openjdk.jmh.Main'
-apply plugin: 'com.github.johnrengelman.shadow' // build an uberjar with all benchmarks
 
 // Not published so no need to assemble
 tasks.remove(assemble)
@@ -51,10 +48,8 @@ compileJava.options.compilerArgs << "-Xlint:-cast,-deprecation,-rawtypes,-try,-u
 // needs to be added separately otherwise Gradle will quote it and javac will fail
 compileJava.options.compilerArgs.addAll(["-processor", "org.openjdk.jmh.generators.BenchmarkProcessor"])
 
-forbiddenApis {
-    // classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
-    ignoreFailures = true
-}
+// classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
+forbiddenApisMain.enabled = false
 
 // No licenses for our benchmark deps (we don't ship benchmarks)
 dependencyLicenses.enabled = false
@@ -70,20 +65,3 @@ thirdPartyAudit.excludes = [
         'org.openjdk.jmh.profile.HotspotRuntimeProfiler',
         'org.openjdk.jmh.util.Utils'
 ]
-
-runShadow {
-    executable = new File(project.runtimeJavaHome, 'bin/java')
-}
-
-// alias the shadowJar and runShadow tasks to abstract from the concrete plugin that we are using and provide a more consistent interface
-task jmhJar(
-        dependsOn: shadowJar,
-        description: 'Generates an uberjar with the microbenchmarks and all dependencies',
-        group: 'Benchmark'
-)
-
-task jmh(
-        dependsOn: runShadow,
-        description: 'Runs all microbenchmarks',
-        group: 'Benchmark'
-)

benchmarks/src/main/resources/log4j2.properties

@@ -1,7 +1,7 @@
 appender.console.type = Console
 appender.console.name = console
 appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%m%n
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %m%n
 
 # Do not log at all if it is not really critical - we're in a benchmark
 rootLogger.level = error

buildSrc/build.gradle

@@ -183,4 +183,12 @@ if (project != rootProject) {
     testClass = 'org.elasticsearch.gradle.test.GradleUnitTestCase'
     integTestClass = 'org.elasticsearch.gradle.test.GradleIntegrationTestCase'
   }
+
+  /*
+   * We alread configure publication and we don't need or want this one that
+   * comes from the java-gradle-plugin.
+   */
+  afterEvaluate {
+    generatePomFileForPluginMavenPublication.enabled = false
+  }
 }

buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy

@@ -405,6 +405,10 @@ class BuildPlugin implements Plugin<Project> {
             repos.mavenLocal()
         }
         repos.mavenCentral()
+        repos.maven {
+            name "elastic"
+            url "https://artifacts.elastic.co/maven"
+        }
         String luceneVersion = VersionProperties.lucene
         if (luceneVersion.contains('-snapshot')) {
             // extract the revision number from the version with a regex matcher
@@ -778,11 +782,16 @@ class BuildPlugin implements Plugin<Project> {
                     systemProperty property.getKey(), property.getValue()
                 }
             }
+
+            // TODO: remove this once joda time is removed from scriptin in 7.0
+            systemProperty 'es.scripting.use_java_time', 'true'
+
             // Set the system keystore/truststore password if we're running tests in a FIPS-140 JVM
             if (project.inFipsJvm) {
                 systemProperty 'javax.net.ssl.trustStorePassword', 'password'
                 systemProperty 'javax.net.ssl.keyStorePassword', 'password'
             }
+
             boolean assertionsEnabled = Boolean.parseBoolean(System.getProperty('tests.asserts', 'true'))
             enableSystemAssertions assertionsEnabled
             enableAssertions assertionsEnabled

buildSrc/version.properties

@@ -1,5 +1,5 @@
 elasticsearch     = 6.5.0
-lucene            = 7.5.0-snapshot-b9e064b935
+lucene            = 7.5.0-snapshot-608f0277b0
 
 # optional dependencies
 spatial4j         = 0.7

client/benchmark/README.md

@@ -2,43 +2,54 @@
 
 1. Build `client-benchmark-noop-api-plugin` with `gradle :client:client-benchmark-noop-api-plugin:assemble`
 2. Install it on the target host with `bin/elasticsearch-plugin install file:///full/path/to/client-benchmark-noop-api-plugin.zip`
-3. Start Elasticsearch on the target host (ideally *not* on the same machine)
-4. Build an uberjar with `gradle :client:benchmark:shadowJar` and execute it.
+3. Start Elasticsearch on the target host (ideally *not* on the machine
+that runs the benchmarks)
+4. Run the benchmark with
+```
+./gradlew -p client/benchmark run --args ' params go here'
+```
 
-Repeat all steps above for the other benchmark candidate.
+Everything in the `'` gets sent on the command line to JMH. The leading ` `
+inside the `'`s is important. Without it parameters are sometimes sent to
+gradle.
+
+See below for some example invocations.
 
 ### Example benchmark
 
 In general, you should define a few GC-related settings `-Xms8192M -Xmx8192M -XX:+UseConcMarkSweepGC -verbose:gc -XX:+PrintGCDetails` and keep an eye on GC activity. You can also define `-XX:+PrintCompilation` to see JIT activity.
 
 #### Bulk indexing
 
-Download benchmark data from http://benchmarks.elastic.co/corpora/geonames/documents.json.bz2 and decompress them.
+Download benchmark data from http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames and decompress them.
 
-Example command line parameters:
+Example invocation:
 
 ```
-rest bulk 192.168.2.2 ./documents.json geonames type 8647880 5000
+wget http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents-2.json.bz2
+bzip2 -d documents-2.json.bz2
+mv documents-2.json client/benchmark/build
+gradlew -p client/benchmark run --args ' rest bulk localhost build/documents-2.json geonames type 8647880 5000'
 ```
 
-The parameters are in order:
+The parameters are all in the `'`s and are in order:
 
 * Client type: Use either "rest" or "transport"
 * Benchmark type: Use either "bulk" or "search"
 * Benchmark target host IP (the host where Elasticsearch is running)
 * full path to the file that should be bulk indexed
 * name of the index
-* name of the (sole) type in the index 
+* name of the (sole) type in the index
 * number of documents in the file
 * bulk size
 
 
-#### Bulk indexing
+#### Search
 
-Example command line parameters:
+Example invocation:
 
 ```
-rest search 192.168.2.2 geonames "{ \"query\": { \"match_phrase\": { \"name\": \"Sankt Georgen\" } } }\"" 500,1000,1100,1200
+gradlew -p client/benchmark run --args ' rest search localhost geonames {"query":{"match_phrase":{"name":"Sankt Georgen"}}} 500,1000,1100,1200'
 ```
 
 The parameters are in order:
@@ -49,5 +60,3 @@ The parameters are in order:
 * name of the index
 * a search request body (remember to escape double quotes). The `TransportClientBenchmark` uses `QueryBuilders.wrapperQuery()` internally which automatically adds a root key `query`, so it must not be present in the command line parameter.
 * A comma-separated list of target throughput rates
-
-

client/benchmark/build.gradle

@@ -18,9 +18,6 @@
  */
 
 apply plugin: 'elasticsearch.build'
-// build an uberjar with all benchmarks
-apply plugin: 'com.github.johnrengelman.shadow'
-// have the shadow plugin provide the runShadow task
 apply plugin: 'application'
 
 group = 'org.elasticsearch.client'
@@ -32,7 +29,6 @@ build.dependsOn.remove('assemble')
 archivesBaseName = 'client-benchmarks'
 mainClassName = 'org.elasticsearch.client.benchmark.BenchmarkMain'
 
-
 // never try to invoke tests on the benchmark project - there aren't any
 check.dependsOn.remove(test)
 // explicitly override the test task too in case somebody invokes 'gradle test' so it won't trip

client/benchmark/src/main/resources/log4j2.properties

@@ -1,7 +1,7 @@
 appender.console.type = Console
 appender.console.name = console
 appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%m%n
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %m%n
 
 rootLogger.level = info
 rootLogger.appenderRef.console.ref = console

client/rest-high-level/src/main/java/org/elasticsearch/client/LicenseClient.java

@@ -0,0 +1,66 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.client;
+
+import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.protocol.xpack.license.PutLicenseRequest;
+import org.elasticsearch.protocol.xpack.license.PutLicenseResponse;
+
+import java.io.IOException;
+
+import static java.util.Collections.emptySet;
+
+/**
+ * A wrapper for the {@link RestHighLevelClient} that provides methods for
+ * accessing the Elastic License-related methods
+ * <p>
+ * See the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/licensing-apis.html">
+ * X-Pack Licensing APIs on elastic.co</a> for more information.
+ */
+public class LicenseClient {
+
+    private final RestHighLevelClient restHighLevelClient;
+
+    LicenseClient(RestHighLevelClient restHighLevelClient) {
+        this.restHighLevelClient = restHighLevelClient;
+    }
+
+    /**
+     * Updates license for the cluster.
+     * @param options the request options (e.g. headers), use {@link RequestOptions#DEFAULT} if nothing needs to be customized
+     * @return the response
+     * @throws IOException in case there is a problem sending the request or parsing back the response
+     */
+    public PutLicenseResponse putLicense(PutLicenseRequest request, RequestOptions options) throws IOException {
+        return restHighLevelClient.performRequestAndParseEntity(request, RequestConverters::putLicense, options,
+            PutLicenseResponse::fromXContent, emptySet());
+    }
+
+    /**
+     * Asynchronously updates license for the cluster cluster.
+     * @param options the request options (e.g. headers), use {@link RequestOptions#DEFAULT} if nothing needs to be customized
+     * @param listener the listener to be notified upon request completion
+     */
+    public void putLicenseAsync(PutLicenseRequest request, RequestOptions options, ActionListener<PutLicenseResponse> listener) {
+        restHighLevelClient.performRequestAsyncAndParseEntity(request, RequestConverters::putLicense, options,
+            PutLicenseResponse::fromXContent, listener, emptySet());
+    }
+
+}