Merge remote-tracking branch 'upstream/master' into add_jmx_kafka_tests

.github/ISSUE_TEMPLATE/module-checklist.md

@@ -21,8 +21,7 @@ For a metricset to go GA, the following criterias should be met:
 * [ ] Fields follow [ECS](https://github.com/elastic/ecs) and [naming conventions](https://www.elastic.co/guide/en/beats/devguide/master/event-conventions.html)
 * [ ] Dashboards exists (if applicable)
 * [ ] Kibana Home Tutorial (if applicable)
- * [ ] Open issue in [EUI repo](https://github.com/elastic/eui) to add [icon for module](https://elastic.github.io/eui/#/display/icons) if not already exists.
- * [ ] Open PR against Kibana repo with tutorial. Examples can be found [here](https://github.com/elastic/kibana/tree/master/src/legacy/core_plugins/kibana/server/tutorials).
+ * Open PR against Kibana repo with tutorial. Examples can be found [here](https://github.com/elastic/kibana/tree/master/src/legacy/core_plugins/kibana/server/tutorials).
 
 ## Filebeat module
 

CHANGELOG.next.asciidoc

@@ -23,6 +23,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - By default, all Beats-created files and folders will have a umask of 0027 (on POSIX systems). {pull}14119[14119]
 - Adding new `Enterprise` license type to the licenser. {issue}14246[14246]
 - Change wording when we fail to load a CA file to the cert pool. {issue}14309[14309]
+- Allow Metricbeat's beat module to read monitoring information over a named pipe or unix domain socket. {pull}14558[14558]
 
 *Auditbeat*
 
@@ -196,6 +197,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix handling multiline log entries in nginx module. {issue}14349[14349] {pull}14499[14499]
 - Fix parsing of Elasticsearch node name by `elasticsearch/slowlog` fileset. {pull}14547[14547]
 - cisco/asa fileset: Fix parsing of 302021 message code. {pull}14519[14519]
+- Fix filebeat azure dashboards, event category should be `Alert`. {pull}14668[14668]
 
 *Heartbeat*
 
@@ -251,6 +253,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Change kubernetes.event.message to text. {pull}13964[13964]
 - Fix performance counter values for windows/perfmon metricset. {issue}14036[14036] {pull}14039[14039]
 - Add FailOnRequired when applying schema and fix metric names in mongodb metrics metricset. {pull}14143[14143]
+- Change `server_status_path` default setting for nginx module {issue}13806[13806] {pull}14099[14099]
 - Convert increments of 100 nanoseconds/ticks to milliseconds for WriteTime and ReadTime in diskio metricset (Windows) for consistency. {issue}14233[14233]
 - Limit some of the error messages to the logs only {issue}14317[14317] {pull}14327[14327]
 - Convert indexed ms-since-epoch timestamp fields in `elasticsearch/ml_job` metricset to ints from float64s. {issue}14220[14220] {pull}14222[14222]
@@ -276,6 +279,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 *Winlogbeat*
 
 - Fix data race affecting config validation at startup. {issue}13005[13005]
+- Set host.name to computername in Windows event logs & sysmon. Requires {pull}14407[14407] in libbeat to work {issue}13706[13706]
 
 *Functionbeat*
 
@@ -411,8 +415,10 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add support for http hostname in nginx filebeat module. {pull}14505[14505]
 - Add attack_pattern_kql field to MISP threat indicators. {pull}14470[14470]
 - Add fileset to the Zeek module for the intel.log. {pull}14404[14404]
+- Add vpc flow log fileset to AWS module. {issue}13880[13880] {pull}14345[14345]
 - New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. {pull}14553[14553]
 - Add document for Filebeat input httpjson. {pull}14602[14602]
+- Add more configuration options to the Netflow module. {pull}14628{14628}
 
 *Heartbeat*
 - Add non-privileged icmp on linux and darwin(mac). {pull}13795[13795] {issue}11498[11498]
@@ -499,6 +505,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add more TCP states to Metricbeat system socket_summary. {pull}14347[14347]
 - Add Kafka JMX metricsets. {pull}14330[14330]
 - Add metrics to envoyproxy server metricset and support for envoy proxy 1.12. {pull}14416[14416] {issue}13642[13642]
+- Release kubernetes modules `controllermanager`, `scheduler`, `proxy`, `state_cronjob` and `state_resourcequota` as GA. {pull}14584[14584]
 - Add module for ActiveMQ. {pull}14580[14580]
 
 *Packetbeat*

NOTICE.txt

@@ -480,40 +480,6 @@ Copyright 2014 CoreOS, Inc
 This product includes software developed at CoreOS, Inc.
 (http://www.coreos.com/).
 
---------------------------------------------------------------------
-Dependency: github.com/DataDog/zstd
-Revision: 2bf71ec4836011b92dc78df3b9ace6b40e65f7df
-License type (autodetected): BSD-3-Clause
-./vendor/github.com/DataDog/zstd/LICENSE:
---------------------------------------------------------------------
-Simplified BSD License
-
-Copyright (c) 2016, Datadog <info@datadoghq.com>
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
- * Redistributions of source code must retain the above copyright notice,
- this list of conditions and the following disclaimer.
- * Redistributions in binary form must reproduce the above copyright notice,
- this list of conditions and the following disclaimer in the documentation
- and/or other materials provided with the distribution.
- * Neither the name of the copyright holder nor the names of its contributors
- may be used to endorse or promote products derived from this software
- without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
 --------------------------------------------------------------------
 Dependency: github.com/davecgh/go-spew
 Version: v1.1.0
@@ -3864,6 +3830,41 @@ License type (autodetected): BSD-3-Clause
 ./vendor/github.com/klauspost/compress/LICENSE:
 --------------------------------------------------------------------
 Copyright (c) 2012 The Go Authors. All rights reserved.
+Copyright (c) 2019 Klaus Post. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+ * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+--------------------------------------------------------------------
+Dependency: github.com/klauspost/compress/snappy
+Revision: c099ac9f21ddbdc8fda7992ce71fae347bf1df88
+License type (autodetected): BSD-3-Clause
+./vendor/github.com/klauspost/compress/snappy/LICENSE:
+--------------------------------------------------------------------
+Copyright (c) 2011 The Snappy-Go Authors. All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are
@@ -3891,6 +3892,35 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+--------------------------------------------------------------------
+Dependency: github.com/klauspost/compress/zstd/internal/xxhash
+Revision: c099ac9f21ddbdc8fda7992ce71fae347bf1df88
+License type (autodetected): MIT
+./vendor/github.com/klauspost/compress/zstd/internal/xxhash/LICENSE.txt:
+--------------------------------------------------------------------
+Copyright (c) 2016 Caleb Spare
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
 --------------------------------------------------------------------
 Dependency: github.com/klauspost/cpuid
 Revision: 09cded8978dc9e80714c4d85b0322337b0a1e5e0
@@ -4798,8 +4828,8 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 --------------------------------------------------------------------
 Dependency: github.com/Shopify/sarama
-Version: v1.23.1
-Revision: 46c83074a05474240f9620fb7c70fb0d80ca401a
+Version: v1.24.1-elastic
+Revision: 355d120d09706d85bc6d2a84287be7fbd13ef453
 License type (autodetected): MIT
 ./vendor/github.com/Shopify/sarama/LICENSE:
 --------------------------------------------------------------------

auditbeat/docs/auditbeat-filtering.asciidoc

@@ -1,6 +1,6 @@
 [[filtering-and-enhancing-data]]
 == Filter and enhance the exported data
 
-include::{libbeat-dir}/docs/processors.asciidoc[]
+include::{libbeat-dir}/processors.asciidoc[]
 
-include::{libbeat-dir}/docs/processors-using.asciidoc[]
+include::{libbeat-dir}/processors-using.asciidoc[]

auditbeat/docs/auditbeat-general-options.asciidoc

@@ -4,4 +4,4 @@
 You can specify settings in the +{beatname_lc}.yml+ config file to control the
 general behavior of {beatname_uc}.
 
-include::{libbeat-dir}/docs/generalconfig.asciidoc[]
+include::{libbeat-dir}/generalconfig.asciidoc[]

auditbeat/docs/configuring-howto.asciidoc

@@ -51,38 +51,38 @@ include::./auditbeat-general-options.asciidoc[]
 
 include::./reload-configuration.asciidoc[]
 
-include::{libbeat-dir}/docs/queueconfig.asciidoc[]
+include::{libbeat-dir}/queueconfig.asciidoc[]
 
-include::{libbeat-dir}/docs/outputconfig.asciidoc[]
+include::{libbeat-dir}/outputconfig.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-ilm.asciidoc[]
+include::{libbeat-dir}/shared-ilm.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-ssl-config.asciidoc[]
+include::{libbeat-dir}/shared-ssl-config.asciidoc[]
 
 include::./auditbeat-filtering.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
+include::{libbeat-dir}/shared-config-ingest.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+include::{libbeat-dir}/shared-geoip.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
+include::{libbeat-dir}/shared-path-config.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]
+include::{libbeat-dir}/shared-kibana-config.asciidoc[]
 
-include::{libbeat-dir}/docs/setup-config.asciidoc[]
+include::{libbeat-dir}/setup-config.asciidoc[]
 
-include::{libbeat-dir}/docs/loggingconfig.asciidoc[]
+include::{libbeat-dir}/loggingconfig.asciidoc[]
 
 :standalone:
-include::{libbeat-dir}/docs/shared-env-vars.asciidoc[]
+include::{libbeat-dir}/shared-env-vars.asciidoc[]
 :standalone!:
 
 :standalone:
-include::{libbeat-dir}/docs/yaml.asciidoc[]
+include::{libbeat-dir}/yaml.asciidoc[]
 :standalone!:
 
-include::{libbeat-dir}/docs/regexp.asciidoc[]
+include::{libbeat-dir}/regexp.asciidoc[]
 
-include::{libbeat-dir}/docs/http-endpoint.asciidoc[]
+include::{libbeat-dir}/http-endpoint.asciidoc[]
 
-include::{libbeat-dir}/docs/reference-yml.asciidoc[]
+include::{libbeat-dir}/reference-yml.asciidoc[]

auditbeat/docs/faq.asciidoc

@@ -7,6 +7,6 @@ https://discuss.elastic.co/c/beats/{beatname_lc}[{beatname_uc} discussion forum]
 
 include::./faq-ulimit.asciidoc[]
 
-include::{libbeat-dir}/docs/faq-limit-bandwidth.asciidoc[]
+include::{libbeat-dir}/faq-limit-bandwidth.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-faq.asciidoc[]
+include::{libbeat-dir}/shared-faq.asciidoc[]

auditbeat/docs/getting-started.asciidoc

@@ -1,7 +1,7 @@
 [id="{beatname_lc}-getting-started"]
 == Getting started with {beatname_uc}
 
-include::{libbeat-dir}/docs/shared-getting-started-intro.asciidoc[]
+include::{libbeat-dir}/shared-getting-started-intro.asciidoc[]
 
 * <<{beatname_lc}-installation>>
 * <<{beatname_lc}-configuration>>
@@ -16,7 +16,7 @@ include::{libbeat-dir}/docs/shared-getting-started-intro.asciidoc[]
 
 Install {beatname_uc} on all the servers you want to monitor.
 
-include::{libbeat-dir}/docs/shared-download-and-install.asciidoc[]
+include::{libbeat-dir}/shared-download-and-install.asciidoc[]
 
 [[deb]]
 *deb:*
@@ -75,7 +75,7 @@ tar xzvf {beatname_lc}-{version}-darwin-x86_64.tar.gz
 
 endif::[]
 
-include::{libbeat-dir}/docs/shared-brew-install.asciidoc[]
+include::{libbeat-dir}/shared-brew-install.asciidoc[]
 
 [[linux]]
 *linux:*
@@ -162,7 +162,7 @@ For more information about these options, see
 [id="{beatname_lc}-configuration"]
 === Step 2: Configure {beatname_uc}
 
-include::{libbeat-dir}/docs/shared-configuring.asciidoc[]
+include::{libbeat-dir}/shared-configuring.asciidoc[]
 
 To configure {beatname_uc}:
 
@@ -192,25 +192,25 @@ If you accept the default configuration without specifying additional modules,
 +
 See <<configuring-howto-{beatname_lc}>> for more details about configuring modules.
 
-include::{libbeat-dir}/docs/step-configure-output.asciidoc[]
+include::{libbeat-dir}/step-configure-output.asciidoc[]
 
-include::{libbeat-dir}/docs/step-configure-kibana-endpoint.asciidoc[]
+include::{libbeat-dir}/step-configure-kibana-endpoint.asciidoc[]
 
-include::{libbeat-dir}/docs/step-configure-credentials.asciidoc[]
+include::{libbeat-dir}/step-configure-credentials.asciidoc[]
 
-include::{libbeat-dir}/docs/step-test-config.asciidoc[]
+include::{libbeat-dir}/step-test-config.asciidoc[]
 
-include::{libbeat-dir}/docs/step-look-at-config.asciidoc[]
+include::{libbeat-dir}/step-look-at-config.asciidoc[]
 
 [id="{beatname_lc}-template"]
 === Step 3: Load the index template in {es}
 
-include::{libbeat-dir}/docs/shared-template-load.asciidoc[]
+include::{libbeat-dir}/shared-template-load.asciidoc[]
 
 [[load-kibana-dashboards]]
 === Step 4: Set up the {kib} dashboards
 
-include::{libbeat-dir}/docs/dashboards.asciidoc[]
+include::{libbeat-dir}/dashboards.asciidoc[]
 
 [id="{beatname_lc}-starting"]
 === Step 5: Start {beatname_uc}
@@ -246,7 +246,7 @@ in the _Beats Platform Reference_.
 If you see a warning about too many open files, you need to increase the
 `ulimit`. See the <<ulimit,FAQ>> for more details.
 
-include::{libbeat-dir}/docs/shared-brew-run.asciidoc[]
+include::{libbeat-dir}/shared-brew-run.asciidoc[]
 
 *win:*
 
@@ -280,7 +280,7 @@ To make it easier for you to start auditing the activities of users and
 processes on your system, we have created example {beatname_uc} dashboards.
 You loaded the dashboards earlier when you ran the `setup` command.
 
-include::{libbeat-dir}/docs/opendashboards.asciidoc[]
+include::{libbeat-dir}/opendashboards.asciidoc[]
 
 The dashboards are provided as examples. We recommend that you
 {kibana-ref}/dashboard.html[customize] them to meet your needs.

auditbeat/docs/index.asciidoc

@@ -1,8 +1,8 @@
 = Auditbeat Reference
 
-:libbeat-dir: {docdir}/../../libbeat
+:libbeat-dir: {docdir}/../../libbeat/docs
 
-include::{libbeat-dir}/docs/version.asciidoc[]
+include::{libbeat-dir}/version.asciidoc[]
 
 include::{asciidoc-dir}/../../shared/versions/stack/{source_branch}.asciidoc[]
 
@@ -25,13 +25,13 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :no_script_processor:
 :no_timestamp_processor:
 
-include::{libbeat-dir}/docs/shared-beats-attributes.asciidoc[]
+include::{libbeat-dir}/shared-beats-attributes.asciidoc[]
 
 include::./overview.asciidoc[]
 
 include::./getting-started.asciidoc[]
 
-include::{libbeat-dir}/docs/repositories.asciidoc[]
+include::{libbeat-dir}/repositories.asciidoc[]
 
 include::./setting-up-running.asciidoc[]
 
@@ -43,13 +43,13 @@ include::./modules.asciidoc[]
 
 include::./fields.asciidoc[]
 
-include::{libbeat-dir}/docs/monitoring/monitoring-beats.asciidoc[]
+include::{libbeat-dir}/monitoring/monitoring-beats.asciidoc[]
 
-include::{libbeat-dir}/docs/shared-securing-beat.asciidoc[]
+include::{libbeat-dir}/shared-securing-beat.asciidoc[]
 
 include::./troubleshooting.asciidoc[]
 
 include::./faq.asciidoc[]
 
-include::{libbeat-dir}/docs/contributing-to-beats.asciidoc[]
+include::{libbeat-dir}/contributing-to-beats.asciidoc[]