Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i5354: Harmonize field names between Packetbeat HTTP and Metricbeat Module #5469

Closed
wants to merge 305 commits into from

Conversation

christiangalsterer
Copy link
Contributor

Should also get merged to master. Shall I create a separate PR for this?

urso and others added 30 commits August 9, 2017 14:14
* Fix monitor.name being empty by default

* Update dashboards to filter by monitor.type
…c#4858)

* Allow to use -e to provide a list of directories which should be excluded in case of hard code it
* Allow to use -s to define NOTICE files which should be skipped
* Set the description of the notice file based on the -b flag
Setting `index: false` for a field was not possible so far. This adds this possibility. For more details in `index` see: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/mapping-index.html

Additional changes:

* `enable` and `dynamic` are only available for objects. The functionality was move to object. https://www.elastic.co/guide/en/elasticsearch/reference/5.5/enabled.html
* Add support for `enable` and `index` to the docs script so it is visible in the docs which fields are indexed and which ones not
Quite a few of our tests fail because the registry does not exist yet when checked. This PR changes the get_registry function to wait until the registry exists and then return it. This should make some of the tests more stable.
This should prevent the test_close_renamed test to be flaky on appveyor. Sometimes the registry was requested before it was written. See https://ci.appveyor.com/project/elastic-beats/beats/build/12109/job/g5lxn7k6fcgsi5ka as example.
* Added breaking changes doc for 6.0

Part of elastic#4737, but there are some TODOs left.

* Add section for multiple outputs

Also, addressed commits and removed the 5.0 breaking changes.
The command that collected the configurations, used something like
this to list the modules:

   ls module | grep -v .go

Problem was, `mongodb` matched the `.go` grep, so it was excluded :)

This PR uses `find` to list the directories.
Currently the dev-guide is not build as part of `make docs`. This adjusments to the scripts and Makefile add the dev-guide and offer more flexibility for docs build also for community beats.

The script was also moved out of libbeat to the global script directory as it does not directly belong to libbeat.
This is meant to make it easy for us to test with security enabled.
Fix MySQL CRUD expression
- iostat.read_request_merged_per_sec
- iostat.write_request_merged_per_sec
- iostat.read_request_per_sec
- iostat.write_request_per_sec
- iostat.read_byte_per_sec
- iostat.write_byte_per_sec
- iostat.avg_request_size
- iostat.avg_queue_size
- iostat.await
- iostat.service_time
- iostat.busy

these metrics has same meaninig of `iostat`. docs and data.json are both updated.
this feature is only avaliable on linux.
* Unmark setup as deprecated

We decided to keep `-setup` as a shorthand for setup + run

* Update `test output` help message
…e as for Elasticsearch (elastic#4867)

- Raise an error in case there are no dashboards to be imported
…astic#4880)

* Remove all references to flush_interval

The outputs setting flush_interval has been removed in the past. Remove
this setting from reference configuration and docs as well.

* Fail if removed flush_interval is configured
This adds a new metricset that reports the system uptime. The metricset
is enabled by default with a period of 15m.

The metricset reports one field `system.uptime.duration.ms` that is the system's
uptime reported in milliseconds. In Kibana the field is formatted as a `duration`
and shown in human-readable format (e.g. "8 days").

Closes elastic#4848
Type of array items needs to be defined when creating a mapping for arrays. For dynamic mapping ES uses the type of the first array item. For pre-defined mapping use ObjectType information set in the fields.yml if available.
* Add docs for new subcommands

* Add changes from review
…and syntax (elastic#4841)

* Update getting started to include info about credentials and new command syntax

* Add fixes from review
Move configuration options from module page to their respective metricset pages.

- Add `process.include_cpu_ticks` to reference config file (it was missing).
- Rename Darwin to macOS in list of supported operating systems.
- Add docs for `core.metrics` and `cpu.metrics`.
tsg and others added 9 commits October 29, 2017 23:42
There was a length check missing.
Fix typo of split_cookie option packetbeat
* Catch duplicate keys in integration tests

This enabled duplicate key detection when we read the results in
the system tests. This makes sure that if we ever have a duplicate key
like in elastic#5440 we will notice.

Part of elastic#5437.
* Make python tests not executables

Turns out that if a `test_*.py` file is executable, nosetests
doesn't pick it up, so the tests are not executed on Linux/Mac.
This was the case with two files in Filebeat.

This adds a `make check` check to look for such files, so this
doesn't happen in the future.
* Move sentence to correct location

* Adds warning about deleting the index (fixes elastic#5083)

* Clarify settings for JSON parsing (fixes elastic#4391)

* Fix misc issues
tsg and others added 7 commits November 1, 2017 11:21
* registry.Start must report errors

* Fix race condition in harvester limit

This fixes a race where the size of registry's internal map was
used to count the number of harvesters, but this map is handled
in a different goroutine.

Closes elastic#5458

* Re-enable flaky test

* Updated CHANGELOG

* Fix imports ordering
The default namespace should be populated by the Kubernetes client. In the case
of in-cluster config, the namespace will be read from the
/var/run/secrets/kubernetes.io/serviceaccount/namespace file. (Note that this
assumes the ServiceAccount admission controller is in use.) In the case of
parsing a k8s configuration file, the default namespace is set to the client
default of "default".

Minor spelling correction of Kubernetes and some additional error information
was changed as well.

Closes elastic#5378
The windows service metricset for Metricbeat collects state information and uptime metrics from all services in the system. If there are services or states that you are not interested in having the metricset report then you can add a `drop_event` processor.

Closes elastic#5256
pull request elastic#5458 was merged with some outstanding changes
* Add `cloud.id` support to Kubernetes manifests

* Always make update *-kubernetes.yaml files
@ruflin
Copy link
Contributor

ruflin commented Nov 6, 2017

LGTM. Could you open this PR against master instead of 6.0 so we can decide afterwards to which branches we backport it?

You need to run make fmt to make CI happy.

Sterchele Nicolas and others added 4 commits November 6, 2017 12:00
…tic#5431)

Change made on GenCertsForIPIfMIssing function:
- Rename the function to GenCertForTestingPurpose
- Replace ip parameter by host, can be an ip or a dns name
- Add password parameter to generate encrypted private key
* Fix ML jobs setup for dynamic modules

Modules from `modules.d` were ignored by both `setup` command and
`--setup` flag.

Fixes elastic#5504

* Fix test
* Fix relative paths in prospectors definitions

If a relative path was used in a prospector definition, it could happen
that when loading the state from the registry, the path didn't match the
pattern, which made it ignore the saved state. Fixes elastic#5442.

Additionally, this PR cleans up some code related to the `recursive_glob`
setting. The docs used to claim that the feature is disabled by default, but
that wasn't the case and it was impossible to disable it. This change leaves
it enabled by default, but makes it possible to disable it.

* addressed comments
* Add Auditbeat dashboards (overview, sockets, execs)

Add dashboards to view events generated by the Linux audit framework. This PR adds three dashboards:

- Overview - A general dashboard showing a summary of all events.
- Executions - A dashboard showing process executions (`execve` and `execveat`) syscalls.
- Sockets - A dashboard showing information related sockets and remote connectivity (e.g. `bind`, `connect`, `accept`, `recvfrom`).

* Provide example rules to support the dashboards
@christiangalsterer
Copy link
Contributor Author

ok, will do that

@christiangalsterer
Copy link
Contributor Author

New PR available at #5521

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.