Document config file format

libbeat/docs/config-file-format.asciidoc

@@ -0,0 +1,367 @@
+[[config-file-format]]
+== Config file format
+
+Beats config files are based on http://www.yaml.org[YAML], a file format that is
+easier to read and write than other common data formats like XML or JSON.
+
+In beats all YAML files start with an dictionary, an unordered collection of
+name/value pairs. In addition to dictionaries, YAML also supports lists, numbers,
+strings, and many other data types. All members of the same list or dictionary must
+have the same indentation level.
+
+Dictionaries are represented by simple `key: value` pairs all having the same
+indentation level. The colon after `key` must be followed by a space.
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+name: John Doe
+age: 34
+country: Canada
+------------------------------------------------------------------------------
+
+Lists are introduced by dashes `- `. All list members will be lines beginning
+with `- ` at the same indentation level.
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+- Red
+- Green
+- Blue
+------------------------------------------------------------------------------
+
+Lists and dictionaries are used in beats to build structured configurations.
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+filebeat:
+  prospectors:
+    - input_type: log
+      paths:
+        - /var/log/*.log
+      multiline:
+        pattern: '^['
+        match: after
+------------------------------------------------------------------------------
+
+Lists and dictionaries can also be represented in abbreviated form. Abbreviated
+form is somewhat similar to JSON using `{}` for dictionaries and `[]` for lists:
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+person: \{name: "John Doe", age: 34, country: "Canada"}
+colors: ["Red", "Green", "Blue"]
+------------------------------------------------------------------------------
+
+
+[[config-file-format-namespacing]]
+=== Namespacing
+
+All settings are structured using dictionaries and lists. Those are collapsed
+into "namespaced" settings, by creating a setting using the full path of the
+settings name and it's parent structures names, when reading the configuration
+file.
+
+For example this setting:
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+
+output:
+  elasticsearch:
+    index: 'beat-%{+yyyy.MM.dd}'
+
+------------------------------------------------------------------------------
+
+gets collapsed into `output.elasticsearch.index: 'beat-%{+yyyy.MM.dd}'`. The
+full name of a setting is based on all parent structures involved.
+
+Lists create numeric names starting with 0.
+
+For example this filebeat setting:
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+
+filebeat:
+  prospectors:
+    - input_type: log
+
+------------------------------------------------------------------------------
+
+Gets collapsed into `filebeat.prospectors.0.input_type: log`.
+
+Alternatively to using indentation, setting names can be used in collapsed form too.
+
+Note: having two settings with same fully collapsed path is invalid.
+
+Simple filebeat example with partially collapsed setting names and use of compact form:
+
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+
+filebeat.prospectors:
+- input_type: log
+  paths: ["/var/log/*.log"] 
+  multiline.pattern: '^['
+  multiline.match: after
+
+output.elasticsearch.hosts: ["http://localhost:9200"]
+
+------------------------------------------------------------------------------
+
+[[config-file-format-type]]
+=== Config file Data Types
+
+Values of configuration settings are interpreted as required by beats.
+If a value can not be correctly interpreted as the required type - for example a
+string is given when a number is required - the beat will fail to start up.
+
+==== Boolean
+
+Boolean values can be either `true` or `false`. Alternative names for `true` are
+`yes` and `on`. Instead of `false` the values `no` and `off` can be used.
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+enabled: true
+disabled: false
+------------------------------------------------------------------------------
+
+==== Number
+
+Number values require you to enter the number to use without using single or
+double quotes. Some settings only support a restricted number range though.
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+integer: 123
+negative: -1
+float: 5.4
+------------------------------------------------------------------------------
+
+==== String
+
+In YAML[http://www.yaml.org], multiple styles of string definitions are supported:
+ double-quoted, single-quoted, unquoted.
+
+The double-quoted style is specified by surrounding the string with `"`. This
+style provides support for escaping unprintable characters using `\`, but comes
+at the cost of having to escape `\` and `"` characters.
+
+The single-quoted style is specified by surrounding the string with `'`. This
+style supports no escaping (use `''` to quote a single quote). Only printable
+characters can be used when using this form.
+
+Unquoted style requires no quotes, but does not support any escaping plus care
+needs to be taken to not use any symbol that has a speacial meaning in YAML.
+
+Note: Single-quoted style is recommended when defining regular expressions,
+event format strings, windows file paths, or non-alphabetical symbolic characters.
+
+==== Duration
+
+Durations require a numeric value with optional fraction and required unit.
+Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Sometimes features based
+on durations can be disabled by using zero or negative durations.
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+duration1: 2.5s
+duration2: 6h
+duration_disabled: -1s
+------------------------------------------------------------------------------
+
+==== Regular Expression
+
+Regular expressions are special strings getting compiled into regular
+expressions at load time.
+
+As regular expressions and YAML use `\` for escaping
+characters in strings, it's highly recommended to use single quoted strings when
+defining regular expressions. When single quoted strings are used, `\` character
+is not interpreted by YAML parser as escape symbol.
+
+==== Format String (sprintf)
+
+Format strings enable you to refer to event field values creating a string based
+on the current event being processed. Variable expansions are enclosed in
+expansion braces `%{<accessor>:default value}`. Event fields are accessed using
+field references `[fieldname]`. Optional default values can be specified in case the
+field name is missing from the event.
+
+You can also format time stored in the
+`@timestamp` field using the `+FORMAT` syntax where FORMAT is a valid (time
+format)[https://godoc.org/github.com/elastic/beats/libbeat/common/dtfmt].
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+constant-format-string: 'constant string'
+field-format-string: '%{[fieldname]} string'
+format-string-with-date: '%{[fieldname]}-%{+yyyy.MM.dd}'
+------------------------------------------------------------------------------
+
+
+[[config-file-format-env-vars]]
+=== Environment Variables
+
+Beats support use of environment variables in config files to set values that
+need to be configurable during deployment. Environment variable expansion is
+introduced using `${VAR}`, where `VAR` is the name of the environment variable.
+
+Note: Only values can be set using environment variables. Environment variables
+usage in namespace and setting names are not supported.
+
+Variable references are replaced when settings are read by beats. The
+replacement is case-sensitive and occurs after the YAML file itself has been
+parsed. References to undefined variables will lead to errors when dereferenced
+and no default value is specified. To specify a default value, use:
+
+`${VAR:default_value}`
+
+Where `default_value` is the value to use if the environment variable is
+undefined.
+
+If you need to use a literal `${` in your configuration file then you can write
+`$${` to escape the expansion. The `$` symbol can be used to escape other
+characters in the default_value like using `$}` in order to generate a `}`
+character without closing the variable expansion.
+
+After changing the value of an environment variable, the beat needs to be
+restarted to pick up the new value.
+
+[float]
+==== Examples
+
+Here are some examples of configurations that use environment variables
+and what each configuration looks like after replacement:
+
+[options="header"]
+|==================================
+|Config source	       |Environment setting   |Config after replacement
+|`name: ${NAME}`       |`export NAME=elastic` |`name: elastic`
+|`name: ${NAME:beats}` |no setting            |`name: beats`
+|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
+|==================================
+
+[[config-gile-format-refs]]
+=== Reference Variables
+
+Beats settings can reference other settings splicing multiple optionally custom
+named settings into new values. References use the same syntax as
+<<config-file-format-env-vars>> do. Only fully collapsed setting names can be
+referenced to.
+
+For example the filebeat registry file defaults to:
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+filebeat.registry: $\{path.data}/registry
+------------------------------------------------------------------------------
+
+With `path.data` being an implicit config setting, that is overwritable from
+command line, as well as in the configuration file.
+
+Example referencing `es.host` in `output.elasticsearch.hosts`:
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+es.host: '$\{ES_HOST:localhost}'
+
+output.elasticsearch:
+  hosts: ['http://$\{es.host}:9200']
+------------------------------------------------------------------------------
+
+Introducing `es.host`, the host can be overwritten from command line using
+`-E es.host=another-host`.
+
+Plain references, having no default value and are not spliced with other
+references or strings can reference complete namespaces.
+
+These setting with duplicate content:
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+namespace1:
+  subnamespace:
+    host: localhost
+    sleep: 1s
+
+namespace2:
+  subnamespace:
+    host: localhost
+    sleep: 1s
+------------------------------------------------------------------------------
+
+can be rewritten to
+
+["source","yaml",subs="attributes"]
+------------------------------------------------------------------------------
+namespace1: $\{shared}
+namespace2: $\{shared}
+
+shared:
+  subnamespace:
+    host: localhost
+    sleep: 1s
+------------------------------------------------------------------------------
+
+when using plain references.
+
+
+[[config-file-format-cli]]
+=== Command line arguments
+
+Config files to load are set using the `-c` flag on command line. If no flag is
+given, a beat and OS pecific default file path will be assumed.
+
+You can specify multiple configuration files by repeating the `-c` flag. You can
+use this, for example, for setting defaults in a base configuration file, and
+overwrite settings via local configuration files.
+
+In addition to overwriting settings using multiple configuration files,
+individual settings can be overwritten using `-E <setting>=<value>`.
+
+[[config-file-format-tips]]
+=== YAML Tips and Gotchas
+
+When you edit the configuration file, there are a few things that you should know.
+
+[float]
+==== Use Spaces for Indentation
+
+Indentation is meaningful in YAML. Make sure that you use spaces, rather than
+tab characters, to indent sections.
+
+In the default configuration files and in all the examples in the documentation,
+we use 2 spaces per indentation level. We recommend you do the same.
+
+[float]
+==== Look at the Default Config File for Structure
+
+The best way to understand where to define a configuration option is by looking
+at the provided sample configuration files. The configuration files contain most
+of the default configurations that are available per beat. To change a setting,
+simply uncomment the line and change the values.
+
+[float]
+==== Test Your Config File
+
+You can test your configuration file to verify that the structure is valid. Simply run
+your in the foreground with the `-configtest` flag specified. For example:
+
+["source","yaml",subs="attributes,callouts"]
+----------------------------------------------------------------------
+filebeat -c filebeat.yml -configtest
+----------------------------------------------------------------------
+
+You'll see a message if an error in the configuration file is found.
+
+[float]
+==== Wrap Regular Expressions in Single Quotation Marks
+
+If you need to specify a regular expression in a YAML file, it's a good idea to
+wrap the regular expression in single quotation marks to work around YAML's
+tricky rules for string escaping.
+
+For more information about YAML, see http://yaml.org/.

libbeat/docs/index.asciidoc

@@ -22,6 +22,8 @@ include::./installing-beats.asciidoc[]
 
 include::./repositories.asciidoc[]
 
+include::./config-file-format.asciidoc[]
+
 include::./visualizing-data.asciidoc[]
 
 include::./dashboards.asciidoc[]