Fix an issue with a partial enroll, when a user refused to overrides a
local configuration actually the enroll command did already used the
token on the ES cluster, this commit move the confirm in the CM instead
of having it in the Enroll's function and is executed by sending the
token or creating any files on disk.

Fixes: #10150

Moved the definition of `agent.hostname` to fields.common.yml

Missing changelog for #11067

…2 interface (#10962)

This is the first Kubernetes modules which uses the new testing interface too

Fixes a number of issues in the `user` dataset discovered while testing on Fedora 29:

* When calling C functions via cgo, don't rely on errno as an indicator of success - check the return value first.
* Set errno to 0 before calling getpwent(3) or getspent(3).
* Make C function calls thread-safe.
* Address a systemd bug where it can return ENOENT even when there is no error.
* Don't fail on every error.
* Fail system test on WARN/ERROR messages in the log.

These dashboards are now maintained in the new elastic/uptime-contrib repo, so the docs should point there.

TCP checks are not adding URL fields on NXDOMAIN endpoints. This fixes that issue.

It does so by ensuring that URL metadata is added before executing the check, and not during, as done previously.

A side effect of this is that we now perform DNS lookups once per `{hostname,port}` instead of once per `{hostname}`. This is worth the increased simplicity however, as the code would be quite convoluted otherwise, which would put us at risk for more bugs.

Related (but different) 6.x issue: #10777

Fix import formatting

This PR is part of  #10071.

Co-Authored-By: Stamatis Katsaounis <katsaouniss@gmail.com>
Co-Authored-By: Michael Katsoulis <michaelkatsoulis88@gmail.com>

* Import updated ECS 1.0.0 go library.
* Adjust Packetbeat to use int64 for URL ports.
* Re-generate golden files for filebeat and x-pack/filebeat

* Fix fields.yml for sqs from message.* to messages.* and field type
* Change type back to long
* Add format: bytes

After repeatedly running the Elasticsearch module integration test in Metricbeat, I found that sometimes Elasticsearch doesn't get enough time to perform CCR and generate CCR stats. This causes the following error, but only some times:

```
--- FAIL: TestFetch (2.44s)
    --- FAIL: TestFetch/ccr (0.08s)
        elasticsearch_integration_test.go:92:
                Error Trace:    elasticsearch_integration_test.go:92
                Error:          Should NOT be empty, but was []
                Test:           TestFetch/ccr
```

So this PR adds a 300ms sleep to give Elasticsearch enough time to perform CCR and generate CCR stats. After testing various sleep durations, I found that 300ms seemed to be the lowest (round) value I could use that consistently passed this test.

Possibly related: #10866

…1051)

Refer to #10774 for more info

The Monitoring Bulk API (`POST _xpack/monitoring/_bulk`) does not interpret `_type` in the bulk metadata the same was as the regular Bulk API (`POST _bulk`). In the case of the Monitoring Bulk API, `_type` has a special meaning. It does not correspond to the Elasticsearch document `_type` but rather a `type` field within monitoring documents. The `_type` of those monitoring documents gets automatically set to `_doc` by the Monitoring Bulk API.

Therefore, the Beats monitoring code should always send `_type` to the Monitoring Bulk API.

* Add instance.state.name into fields.yml

* Rebase and regenerate fields.yml

* Update changelog

* Fix sqs queue name
* Add function to get queueUrls
* Fix field names from message to messages
* Move looping through each queueURL into a separate function

* Update aws documentation
* Add checks for instance fields in TestFetch
* Add tests for CheckTimestampInArray and FindTimestamp
* Update comment for CheckTimestampInArray
* Change test to table driven
* Adopt CheckTimestampInArray and FindTimestamp in S3 and SQS

This updates the ingest pipeline loading to look at the
pipeline definition file extension and use a JSON or YAML
decoder depending on the type.

The structures `Field` and `Fields` are moved from `common` to the package `mapping`.
This is a step towards moving things from the abused package `common`.
I would like to move fields and other asset related functionality to this package in the future.

At the moment, docker containers are shared between tests. Password test
changes the password as part of its checks, what can affect the
connection of other tests using the same instance, skip this test by
now to avoid flakiness in the rest of tests.

SIGHUP is sent to a process if the user's terminal is disconnected. It is one of the five termination signals. Currently, Beats is going to terminate suddenly, i.e. the log will cut off, no shutdown actions are going to be run, etc.

This change catches SIGHUP as it already does SIGINT/SIGTERM and runs the usual shutdown actions. This can include waiting for a period of time specified using the shutdown_timeout option Filebeat and Packetbeat support.

They were accidentally removed by #11150.

* Update ingest pipelines to use event.timezone

Previously, `beats.timezone` was used, in master it's not available
anymore. However, some ingest pipelines were still using it.

…11232)

Excludes kernel processes on Linux. They don't provide a lot of value, the names are not unique, and they will not be able to use potential future features like process executable hashes.

The script processor executes Javascript code to process an event. The processor uses a pure
Go implementation of ECMAScript 5.1. This can be useful in situations where one of the other
processors doesn’t provide the functionality you need to filter events.

The processor can be configured by embedding Javascript in your configuration file or by pointing
the processor at external file(s). See the included documentation for the full details.

    processors:
    - script:
        type: javascript
        code: >
          function process(event) {
              event.Tag("js");
          }

For observability it can add metrics with a histogram of the execution time and a counter
for the number of exceptions that occur.

There is an optional timeout configuration option to the processor that will
timeout the execution by interrupting the JS code.

Rather than have libbeat force the inclusion of the script processor in all Beats, make each Beat opt into the processor. It is imported by the log processing Beats (filebeat, journalbeat, and winlogbeat).

Benchmark Results

Here's a benchmark to get a rough idea of how long it takes to process an event in the processor's runtime

I updated the benchmark to test each case with and without the timeout enabled.

    BenchmarkBeatEventV0/Put-12                         2000000           707 ns/op
    BenchmarkBeatEventV0/timeout_Put-12                 1000000          1510 ns/op
    BenchmarkBeatEventV0/Object_Put_Key-12              2000000           631 ns/op
    BenchmarkBeatEventV0/timeout_Object_Put_Key-12      1000000          1252 ns/op
    BenchmarkBeatEventV0/Get-12                         2000000           750 ns/op
    BenchmarkBeatEventV0/timeout_Get-12                 1000000          1495 ns/op
    BenchmarkBeatEventV0/Get_Undefined_Key-12           1000000          1039 ns/op
    BenchmarkBeatEventV0/timeout_Get_Undefined_Key-12   1000000          2108 ns/op
    BenchmarkBeatEventV0/fields_get_key-12              2000000          1044 ns/op
    BenchmarkBeatEventV0/timeout_fields_get_key-12      1000000          2051 ns/op
    BenchmarkBeatEventV0/Get_@metadata-12               2000000           750 ns/op
    BenchmarkBeatEventV0/timeout_Get_@metadata-12       1000000          1745 ns/op
    BenchmarkBeatEventV0/Put_@metadata-12               1000000          1048 ns/op
    BenchmarkBeatEventV0/timeout_Put_@metadata-12       500000          2623 ns/op
    BenchmarkBeatEventV0/Delete_@metadata-12            2000000           842 ns/op
    BenchmarkBeatEventV0/timeout_Delete_@metadata-12    1000000          1629 ns/op
    BenchmarkBeatEventV0/Cancel-12                      2000000           759 ns/op
    BenchmarkBeatEventV0/timeout_Cancel-12              1000000          1329 ns/op
    BenchmarkBeatEventV0/Tag-12                         1000000          1189 ns/op
    BenchmarkBeatEventV0/timeout_Tag-12                 1000000          1973 ns/op
    BenchmarkBeatEventV0/AppendTo-12                    2000000           644 ns/op
    BenchmarkBeatEventV0/timeout_AppendTo-12            1000000          1347 ns/op

Setting the PYTHON_EXE environment variable causes new Python virtual environments
to be created using the specified interpreter. Both `make` and `mage` honor the
variable.

For example, `PYTHON_EXE=python2.7 make python-env`.

To help debug #11261.

* Change URLPATH grok pattern to support []

* Update URIPATHWITHBRACKET with the current URIPATH

* Add changelog

…name, etc. in fields.yml (#11134)

* Add Kubernetes missing fields: replica-set.name, statefulset.name and deployment.name, to libbeat add_kubernetes_metadata processor fields.yml and add a few test cases. 
* Remove duplicate fields in metricbeat fields.yml

* Add aws sqs overview dashboard
* Add sqs overview dashboard into documentation

…ce address (#11256)

* Add support for iis 7.5 log with different destination/source address

* update changelog

* Add test with IPv6 ip address

In #9648 the monitor generator script was removed, but its supporting files were not. This commit removes the remaining legacy files.

The repository the documentation was previously linking to (https://github.com/jlevesy/githubbeat) is now inactive, the owner suggests using the following fork instead: https://github.com/josephlewis42/githubbeat

* rabbitmq.node.name was duplicated and alias pointed to itself
* rabbitmq.connection.name was falsely migrated to `rabbitmq.name`

Also adding tests to Filebeat and Metricbeat to ensure this does not happen again in the future.

Closes #11271

Currently we check in python tests that the fields are documented. As by now we have all the fields also available in the go code with the fields.go files it is possible to do this check in the new data tests.

To prevent cyclic imports the data_test.go had to be moved to its own package.

Now by default all modules are imported. Like this we don't have to add each module manually.

Docker containers can have empty ip addresses if they are running in
host network mode or if they are stopped. Collecting lists with empty
addresses can make type mapping to fail when trying to store them as ip
addresses.

Fix #11225

…11288)

This is to make it more clear this is for the migration from 6.7 to 7.0.

for APM, where beatname_lc=apm-server but beat_default_index_prefix=apm

* Fix test failure of auditbeat for s390x

* avoid filebeat test failure on big endian machines

Set a default request timeout of 10 seconds to metricbeat HTTP helpers
so there are less chances of leaking established connections.

Add a connection timeout that defaults to two seconds so connections
fail fast in case of network connectivity problems.

Co-Authored-By: Ioannis Androulidakis <ioannis@arrikto.com>

* Update schema version in monitoring index name

* Update tests

Resolves #9480.

Starting Elasticsearch 7.0.0, Beats should ship their monitoring data to the `_monitoring/bulk` Elasticsearch API endpoint. Prior to 7.0.0, `_xpack/monitoring/_bulk` should be used. This PR implements this version-based conditional logic.

I used Wireshark to look at the ES API endpoints being hit.

Running this PR with ES 8.0.0 or ES 7.0.0, I confirmed that the `POST _monitoring/bulk` endpoint was being hit:

<img width="1436" alt="Screen Shot 2019-03-14 at 10 55 52 AM" src="https://user-images.githubusercontent.com/51061/54380101-ed567780-4647-11e9-8ed1-9b9020bb85d4.png">

And running this PR with ES 6.7.0, I confirmed that the `POST _xpack/monitoring/_bulk` endpoint was being hit:

<img width="1437" alt="Screen Shot 2019-03-14 at 10 56 42 AM" src="https://user-images.githubusercontent.com/51061/54380094-eaf41d80-4647-11e9-8658-d9a6ba14541b.png">

* implement Close() for docker metricsets

Outleters start a goroutine to handle the finalization of filebeat. If the
outleter is closed by other means the goroutine will be kept running
even if it has nothing to do, leaking goroutines.

Stop this goroutine if the outleter is closed.

* treat empty meta as non existent

* test made parallel friendly

* changelog plus typos in beats.go

* perform atomic write of meta.json

Some modules under x-pack had their dashboards images below the OSS beat directory. This PR moves those images under x-pack.

* Check License for basic or better

This reverts commit 9ff8392.

* Adding categorization fields for the system/auth module

This PR adds the following fields for the SSH login events:

* `event.category: authentication`
* `event.action: ssh_login`
* `event.type` either `authentication_success` or `authentication_failure`

The `event.outcome` is currently not quite ECS compliant, but I didn't touch it to
avoid a breaking change.

The PR doesn't attempt to categorize other logs besides the SSH login attempts,
so it's a subset of #9905, but it's what we need for the UI.

* Normalized event.outcome and brought back `system.auth.ssh.event`.

* changelog

After the refactor done in #11032, baseData was not being passed to the
HTTP helper, so authentication was not working.

Added a test to avoid regressions here.

Fix #11351

Signed-off-by: Chris <chrismarkou92@gmail.com>

Block the response only during a time or till the conext is done.
Close body if a request object is returned.

* Migrate mysql/galera_status to ReporterV2

Previous fix (#11006) made Winlogbeat escape CRLF control characters
which are expected in Windows event logs.

Fixes #11328

Closes #10969.

In Kibana as of 7.0, the default query language is no longer Lucene, but KQL (Kibana Query Language).

This PR updates all of the JSON objects to use the new language, and modifies any existing queries to the new syntax (if needed - most of them stayed the same).

* Add JS require with modules

This adds a `require` function to the JS runtime that scripts can call to import
"modules". Some standard modules that are added in this PR are

- `processor` - You can construct beat processors in JS (e.g. `new processor.Dissect({...})`).
- `console` - You can write to beat's logger.
- `path` - You can parse win32 and posix paths.

The `processor` module supports constructing:

- `AddCloudMetadata`
- `AddDockerMetadata`
- `AddHostMetadata`
- `AddKubernetesMetadata`
- `AddLocale`
- `AddProcessMetadata`
- `CommunityID`
- `DecodeJSONFields`
- `Dissect`
- `DNS`

* Add github.com/dop251/goja_nodejs to vendor

Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Chris Mark <chrismarkou92@gmail.com>

Adds `event.category: authentication` and `event.type: authentication_success` (or `authentication_failure`).

A new processor is introduced as part of support for keeping orignal messages. Options and naming follows the convention of other processors.

### `copy_fields`

This processor copies one field to another. Example configuration is below:

```yaml
processors:
- copy_fields:
    fields:
      - from: message
        to: event.original
    fail_on_error: false
    ignore_missing: true
```

* migrate golang module to ReporterV2 error

* Make geoip steps generic and move to libbeat

* Add changes from review

* Add topic to all the beats

* fixed error format to include faulting file,some go lint issues

* changelog updated

* review round 1

* not breaking

Register modules with the script processor.

* wong assumption?

* indicate skip for arrays

* changelog

* make one liner

* not marking as breaking change

* migrate vsphere/datastore to ReporterV2

Add Namespace to prometheus helper mappings

Add test files for MySQL 8, Percona 8 and MariaDB 10.3 to filebeat
mysql module.

Add support for MySQL 8 extra fields in slow log.

Update compatibility notes.

Closes #10257

* Link to infrastructure monitoring docs from Beats docs

* Add links that point to the infrastructure monitoring guide

* Add changes from review

* Add Filebeat coredns module

Fix coredns image in docs.asciidoc for docs build

* Fix docs build again

Update method of common.MapStr doesn't update nested maps
so we were overwriting existing metrics.

Fix fields definition to add wildcard so fields match.

Add testdata.