Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor input docs
Browse files Browse the repository at this point in the history
dedemorton committed Mar 16, 2018
1 parent 1c088e4 commit 7d9d1ed
Showing 30 changed files with 1,127 additions and 705 deletions.
20 changes: 10 additions & 10 deletions docs/devguide/modules-dev-guide.asciidoc
Original file line number Diff line number Diff line change
@@ -176,8 +176,8 @@ There's quite a lot going on in this file, so let's break it down:
element: `"/example/test.log*"`.
* Note that variable values don't have to be strings.
They can be also numbers, objects, or as shown in this example, arrays.
* We will use the `paths` variable to set the prospector
{filebeat}/configuration-filebeat-options.html#prospector-paths[paths]
* We will use the `paths` variable to set the input
{filebeat}/configuration-filebeat-options.html#input-paths[paths]
setting, so "glob" values can be used here.
* Besides the `default` value, the file defines values for particular
operating systems: a default for darwin/OS X/macOS systems and a default for
@@ -186,13 +186,13 @@ There's quite a lot going on in this file, so let's break it down:
Filebeat is executed on the respective OS.

Besides the variable definition, the `manifest.yml` file also contains
references to the ingest pipeline and prospector configuration to use (see next
references to the ingest pipeline and input configuration to use (see next
sections):

[source,yaml]
----
ingest_pipeline: ingest/pipeline.json
prospector: config/testfileset.yml
input: config/testfileset.yml
----

These should point to the respective files from the fileset.
@@ -214,8 +214,8 @@ overridden at runtime.)
[float]
==== config/*.yml

The `config/` folder contains template files that generate Filebeat prospector
configurations. The Filebeat prospectors are primarily responsible for tailing
The `config/` folder contains template files that generate Filebeat input
configurations. The Filebeat inputs are primarily responsible for tailing
files, filtering, and multi-line stitching, so that's what you configure in the
template files.

@@ -233,12 +233,12 @@ exclude_files: [".gz$"]

You'll find this example in the template file that gets generated automatically
when you run `make create-fileset`. In this example, the `paths` variable is
used to construct the `paths` list for the {filebeat}/configuration-filebeat-options.html#prospector-paths[paths] option.
used to construct the `paths` list for the {filebeat}/configuration-filebeat-options.html#input-paths[paths] option.

Any template files that you add to the `config/` folder need to generate a valid
Filebeat prospector configuration in YAML format. The options accepted by the
prospector configuration are documented in the
{filebeat}/configuration-filebeat-options.html[Filebeat Prospectors] section of
Filebeat input configuration in YAML format. The options accepted by the
input configuration are documented in the
{filebeat}/configuration-filebeat-options.html[Filebeat Inputs] section of
the Filebeat documentation.

The template files use the templating language defined by the
8 changes: 4 additions & 4 deletions filebeat/docs/autodiscover-docker-config.asciidoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Filebeat supports templates for both prospectors and modules.
Filebeat supports templates for inputs and modules.

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
@@ -16,9 +16,9 @@ filebeat.autodiscover:
exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines
-------------------------------------------------------------------------------------
This configuration launches a `docker` logs prospector for all containers running an image with `redis` in the name.
This configuration launches a `docker` logs input for all containers running an image with `redis` in the name.
If you are using modules, you can override the default prospector and use the docker prospector instead.
If you are using modules, you can override the default input and use the docker input instead.
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
@@ -32,7 +32,7 @@ filebeat.autodiscover:
config:
- module: redis
log:
prospector:
input:
type: docker
containers.ids:
- "${data.docker.container.id}"
8 changes: 4 additions & 4 deletions filebeat/docs/autodiscover-kubernetes-config.asciidoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Filebeat supports templates for both prospectors and modules.
Filebeat supports templates for inputs and modules.

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
@@ -16,10 +16,10 @@ filebeat.autodiscover:
exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines
-------------------------------------------------------------------------------------
This configuration launches a `docker` logs prospector for all containers of pods running in the Kubernetes namespace
This configuration launches a `docker` logs input for all containers of pods running in the Kubernetes namespace
`kube-system`.
If you are using modules, you can override the default prospector and use the docker prospector instead.
If you are using modules, you can override the default input and use the docker input instead.
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
@@ -33,7 +33,7 @@ filebeat.autodiscover:
config:
- module: redis
log:
prospector:
input:
type: docker
containers.ids:
- "${data.kubernetes.container.id}"
2 changes: 0 additions & 2 deletions filebeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
@@ -47,8 +47,6 @@ include::./filebeat-modules-options.asciidoc[]

include::./filebeat-options.asciidoc[]

include::./multiple-prospectors.asciidoc[]

include::./multiline.asciidoc[]

include::./filebeat-general-options.asciidoc[]
64 changes: 64 additions & 0 deletions filebeat/docs/docker-input.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
:type: docker

[[docker-input]]
=== Docker input

++++
<titleabbrev>Docker</titleabbrev>
++++

experimental[]

Use the `docker` input to read logs from Docker containers.

This input searches for container logs under its path, and parse them into
common message lines, extracting timestamps too. Everything happens before line
filtering, multiline, and JSON decoding, so this input can be used in
combination with those settings.

Example configuration:

// REVIEWERS: The filebeat.reference.yml file does not show a docker input type
// so I'm not sure if the following config is correct. docker only shows up under
// autodiscover.

["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: docker
containers.ids: <1>
- '8b6fe7dc9e067b58476dc57d6986dd96d7100430c5de3b109a99cd56ac655347'
----

<1> `containers.ids` is required. All other settings are optional.

==== Configuration options

The `docker` input supports the following configuration options plus the
<<{type}-input-common-options>> described later.

[[config-container-ids]]
===== `containers.ids`

The list of Docker container IDs to read logs from. Specify
`containers.ids: '*'` to read from all containers.

===== `containers.path`

The base path where Docker logs are located. The default
is `/var/lib/docker/containers`.

===== `containers.stream`

Reads from the specified streams only: `all`, `stdout` or `stderr`. The default
is `all`.

// REVIEWERS: I'm guessing where these settings live in the config
// because it's not clear from the docs or the reference.yml. Would be nice to
// have a realistic example here.

// REVIEWERS: Sounds like I definitely need to add multiline to the docs here.
// which other config options from the harvester are used by the docker input?

[id="{type}-input-common-options"]
include::include/common-input-options.asciidoc[]
2 changes: 1 addition & 1 deletion filebeat/docs/faq.asciidoc
Original file line number Diff line number Diff line change
@@ -74,7 +74,7 @@ You can use <<clean-removed,`clean_removed`>> for files that are removed from di

On Windows, you might have problems renaming or removing files because Filebeat keeps the file handlers open. This can lead to issues with the file rotating system. To avoid this issue, you can use the <<close-removed,`close_removed`>> and <<close-renamed,`close_renamed`>> options together.

IMPORTANT: When you configure these options, files may be closed before the harvester has finished reading the files. If the file cannot be picked up again by the prospector and the harvester hasn't finish reading the file, the missing lines will never be sent to the output.
IMPORTANT: When you configure these options, files may be closed before the harvester has finished reading the files. If the file cannot be picked up again by the input and the harvester hasn't finish reading the file, the missing lines will never be sent to the output.


[float]
10 changes: 5 additions & 5 deletions filebeat/docs/filebeat-filtering.asciidoc
Original file line number Diff line number Diff line change
@@ -6,11 +6,11 @@ you might need to enhance the exported data (for example, by adding metadata).
Filebeat provides a couple of options for filtering and enhancing exported
data.

You can configure each prospector to include or exclude specific lines or files.
This allows you to specify different filtering criteria for each prospector.
You can configure each input to include or exclude specific lines or files.
This allows you to specify different filtering criteria for each input.
To do this, you use the <<include-lines,`include_lines`>>,
<<exclude-lines,`exclude_lines`>>, and <<exclude-files,`exclude_files`>>
options under the `filebeat.prospectors` section of the config file (see
options under the `filebeat.inputs` section of the config file (see
<<configuration-filebeat-options>>). The disadvantage of this approach is that
you need to implement a configuration option for each filtering criteria that
you need.
@@ -67,7 +67,7 @@ The following configuration decodes the inner JSON object:

[source,yaml]
-----------------------------------------------------
filebeat.prospectors:
filebeat.inputs:
- type: log
paths:
- input.json
@@ -94,7 +94,7 @@ The resulting output looks something like this:
"inner": {
"data": "value"
},
"prospector": {
"input": {
"type": "log",
},
"offset": 55,
4 changes: 2 additions & 2 deletions filebeat/docs/filebeat-general-options.asciidoc
Original file line number Diff line number Diff line change
@@ -53,9 +53,9 @@ filebeat.registry_file_permissions: 0600
[float]
==== `config_dir`

The full path to the directory that contains additional prospector configuration files.
The full path to the directory that contains additional input configuration files.
Each configuration file must end with `.yml`. Each config file must also specify the full Filebeat
config hierarchy even though only the prospector part of the file is processed. All global
config hierarchy even though only the `inputs` part of each file is processed. All global
options, such as `registry_file`, are ignored.

The `config_dir` option MUST point to a directory other than the directory where the main Filebeat config file resides.
38 changes: 19 additions & 19 deletions filebeat/docs/filebeat-modules-options.asciidoc
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
[[configuration-filebeat-modules]]
== Specify which modules to run

NOTE: Using Filebeat modules is optional. You may decide to
<<configuration-filebeat-options,set up prospectors manually>> if you are using
NOTE: Using {beatname_uc} modules is optional. You may decide to
<<configuration-filebeat-options,configure inputs manually>> if you are using
a log file type that isn't supported, or you want to use a different setup.

Filebeat <<filebeat-modules,modules>> provide a quick way for you to get started
{beatname_uc} <<filebeat-modules,modules>> provide a quick way for you to get started
processing common log formats. They contain default configurations,
Elasticsearch ingest node pipeline definitions, and Kibana dashboards to help you
implement and deploy a log monitoring solution.

Filebeat provides a few different ways to enable modules. You can:
{beatname_uc} provides a few different ways to enable modules. You can:

* <<enable-modules-d-configs>>
* <<enable-modules-cli>>
@@ -21,9 +21,9 @@ include::../../libbeat/docs/shared-note-file-permissions.asciidoc[]
When you enable modules, you can also
<<specify-variable-settings,specify variable settings>> to change the default
behavior of the modules, and you can specify
<<advanced-settings,advanced settings>> to override prospector settings.
<<advanced-settings,advanced settings>> to override input settings.

Before running Filebeat with modules enabled, make sure you also set up the
Before running {beatname_uc} with modules enabled, make sure you also set up the
environment to use Kibana dashboards. See <<filebeat-modules-quickstart>> for
more information.

@@ -32,7 +32,7 @@ more information.
=== Enable module configs in the `modules.d` directory

The `modules.d` directory contains default configurations for all the modules
available in Filebeat. You can enable or disable specific module configurations
available in {beatname_uc}. You can enable or disable specific module configurations
under `modules.d` by running the
<<modules-command,`modules enable` or `modules disable`>> commands.

@@ -44,7 +44,7 @@ directory, you use:
./filebeat modules enable apache2 mysql
----

Then when you run Filebeat, it loads the corresponding module configurations
Then when you run {beatname_uc}, it loads the corresponding module configurations
specified in the `modules.d` directory (for example, `modules.d/apache2.yml` and
`modules.d/mysql.yml`).

@@ -62,12 +62,12 @@ to specify variable settings. See <<specify-variable-settings>>.

[float]
[[enable-modules-cli]]
=== Enable modules when you run Filebeat
=== Enable modules when you run {beatname_uc}

To enable specific <<filebeat-modules,modules>> when you run Filebeat at the
To enable specific <<filebeat-modules,modules>> when you run {beatname_uc} at the
command line, you can use the `--modules` flag. This approach works well when
you're getting started and want to specify different modules and settings each
time you run Filebeat. Any modules specified at the command line will be loaded
time you run {beatname_uc}. Any modules specified at the command line will be loaded
along with any modules that are enabled in the configuration file or `modules.d`
directory. If there's a conflict, the configuration specified at the command
line is used.
@@ -159,31 +159,31 @@ see the <<filebeat-modules,documentation for the modules>>.
[[advanced-settings]]
=== Advanced settings

Behind the scenes, each module starts a Filebeat prospector. Advanced users
can add or override any prospector settings. For example, you can set
Behind the scenes, each module starts a {beatname_uc} input. Advanced users
can add or override any input settings. For example, you can set
<<close-eof,close_eof>> to `true` in the module configuration:

[source,yaml]
----------------------------------------------------------------------
- module: nginx
access:
prospector:
input:
close_eof: true
----------------------------------------------------------------------

Or at the command line like this:

[source,shell]
----------------------------------------------------------------------
./filebeat -M "nginx.access.prospector.close_eof=true"
./filebeat -M "nginx.access.input.close_eof=true"
----------------------------------------------------------------------


Here you see how to use the `-M` flag along with the `--modules` flag:

[source,shell]
----------------------------------------------------------------------
./filebeat --modules nginx -M "nginx.access.prospector.close_eof=true"
./filebeat --modules nginx -M "nginx.access.input.close_eof=true"
----------------------------------------------------------------------


@@ -193,13 +193,13 @@ modules/filesets at once. For example, the following command enables

[source,shell]
----------------------------------------------------------------------
./filebeat -M "nginx.*.prospector.close_eof=true"
./filebeat -M "nginx.*.input.close_eof=true"
----------------------------------------------------------------------

The following command enables `close_eof` for all prospectors created by any of
The following command enables `close_eof` for all inputs created by any of
the modules:

[source,shell]
----------------------------------------------------------------------
./filebeat -M "*.*.prospector.close_eof=true"
./filebeat -M "*.*.input.close_eof=true"
----------------------------------------------------------------------
Loading

0 comments on commit 7d9d1ed

Please sign in to comment.