Document complex object support in env vars (#4200)

* Document complex ojbect support in env vars

* Fixes from the review

* More changes from the review

filebeat/docs/index.asciidoc

@@ -46,6 +46,7 @@ include::./multiline.asciidoc[]
 
 include::../../libbeat/docs/shared-config-ingest.asciidoc[]
 
+:standalone:
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 include::./multiple-prospectors.asciidoc[]

heartbeat/docs/index.asciidoc

@@ -47,6 +47,7 @@ include::../../libbeat/docs/shared-config-ingest.asciidoc[]
 //points to shared topic because configuring-logstash.asciidoc is just a wrapper
 include::./configuring-logstash.asciidoc[]
 
+:standalone:
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 :standalone:

libbeat/docs/config-file-format.asciidoc

@@ -110,7 +110,7 @@ Simple filebeat example with partially collapsed setting names and use of compac
 
 filebeat.prospectors:
 - type: log
-  paths: ["/var/log/*.log"] 
+  paths: ["/var/log/*.log"]
   multiline.pattern: '^['
   multiline.match: after
 
@@ -213,44 +213,7 @@ format-string-with-date: '%{[fieldname]}-%{+yyyy.MM.dd}'
 [[config-file-format-env-vars]]
 === Environment Variables
 
-Beats support use of environment variables in config files to set values that
-need to be configurable during deployment. Environment variable expansion is
-introduced using `${VAR}`, where `VAR` is the name of the environment variable.
-
-Note: Only values can be set using environment variables. Environment variables
-usage in namespace and setting names are not supported.
-
-Variable references are replaced when settings are read by beats. The
-replacement is case-sensitive and occurs after the YAML file itself has been
-parsed. References to undefined variables will lead to errors when dereferenced
-and no default value is specified. To specify a default value, use:
-
-`${VAR:default_value}`
-
-Where `default_value` is the value to use if the environment variable is
-undefined.
-
-If you need to use a literal `${` in your configuration file then you can write
-`$${` to escape the expansion. The `$` symbol can be used to escape other
-characters in the default_value like using `$}` in order to generate a `}`
-character without closing the variable expansion.
-
-After changing the value of an environment variable, the beat needs to be
-restarted to pick up the new value.
-
-[float]
-==== Examples
-
-Here are some examples of configurations that use environment variables
-and what each configuration looks like after replacement:
-
-[options="header"]
-|==================================
-|Config source	       |Environment setting   |Config after replacement
-|`name: ${NAME}`       |`export NAME=elastic` |`name: elastic`
-|`name: ${NAME:beats}` |no setting            |`name: beats`
-|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
-|==================================
+include::shared-env-vars.asciidoc[]
 
 [[config-gile-format-refs]]
 === Reference Variables
@@ -333,15 +296,15 @@ When installed via an RPM or DEB package, the config file at
 `/etc/{beatname}/{beatname}.yml` will have the proper owner and permissions. The
 file is owned by `root` and has file permissions of `0644` (`-rw-r--r--`).
 
-You may encounter the following errors if your config file fails these checks: 
+You may encounter the following errors if your config file fails these checks:
 
 ["source","sh"]
 --------------------------------------------------------------------------------
 Exiting: error loading config file: config file ("{beatname}.yml") must be
 owned by the beat user (uid=501) or root
 --------------------------------------------------------------------------------
 
-To correct this problem you can use either `chown root {beatname}.yml` or 
+To correct this problem you can use either `chown root {beatname}.yml` or
 `chown 501 {beatname}.yml` to change the owner of the configuration file.
 
 ["source","sh"]
@@ -370,7 +333,43 @@ use this, for example, for setting defaults in a base configuration file, and
 overwrite settings via local configuration files.
 
 In addition to overwriting settings using multiple configuration files,
-individual settings can be overwritten using `-E <setting>=<value>`.
+individual settings can be overwritten using `-E <setting>=<value>`. The
+`<value>` can be either a single value or a complex object, such as a list or
+dictionary.
+
+For example, given the following configuration:
+
+["source","yaml"]
+--------------------------------------------------------------------------------
+output.elasticsearch:
+  hosts: ["http://localhost:9200"]
+  username: username
+  password: password
+--------------------------------------------------------------------------------
+
+You can disable the Elasticsearch output and write all events to the console by
+setting:
+
+["source","sh"]
+--------------------------------------------------------------------------------
+-E output='{elasticsearch.enabled: false, console.pretty: true}'
+--------------------------------------------------------------------------------
+
+Any complex objects that you specify at the command line are merged with the
+original configuration, and the following configuration is passed to the Beat:
+
+["source","yaml"]
+--------------------------------------------------------------------------------
+output.elasticsearch:
+  enabled: false
+  hosts: ["http://localhost:9200"]
+  username: username
+  password: password
+
+output.console:
+  pretty: true
+--------------------------------------------------------------------------------
+
 
 [[config-file-format-tips]]
 === YAML Tips and Gotchas

libbeat/docs/shared-env-vars.asciidoc

@@ -6,13 +6,20 @@
 //// Use the appropriate variables defined in the index.asciidoc file to
 //// resolve Beat names: beatname_uc and beatname_lc.
 //// Use the following include to pull this content into a doc file:
+//// :standalone:
 //// include::../../libbeat/docs/shared-env-vars.asciidoc[]
+//// Specify :standalone: when this file is pulled into and index. When
+//// the file is embedded in another file, do no specify :standalone:
 //////////////////////////////////////////////////////////////////////////
 
+ifdef::standalone[]
+
 [[using-environ-vars]]
 == Using Environment Variables in the Configuration
 
-You can use environment variable references in the +{beatname_lc}.yml+ file to
+endif::[]
+
+You can use environment variable references in the config file to
 set values that need to be configurable during deployment. To do this, use:
 
 `${VAR}`
@@ -22,18 +29,36 @@ Where `VAR` is the name of the environment variable.
 Each variable reference is replaced at startup by the value of the environment
 variable. The replacement is case-sensitive and occurs before the YAML file is
 parsed. References to undefined variables are replaced by empty strings unless
-you specify a default value. To specify a default value, use:
+you specify a default value or custom error text.
+
+To specify a default value, use:
 
 `${VAR:default_value}`
 
 Where `default_value` is the value to use if the environment variable is
 undefined.
 
+To specify custom error text, use:
+
+`${VAR:?error_text}`
+
+Where `error_text` is custom text that will be prepended to the error
+message if the environment variable cannot be expanded.
+
 If you need to use a literal `${` in your configuration file then you can write
 `$${` to escape the expansion.
 
 After changing the value of an environment variable, you need to restart
-{beatname_uc} to pick up the new value.
+the Beat to pick up the new value.
+
+[NOTE]
+==================================
+You can also specify environment variables when you override a config
+setting from the command line by using the `-E` option. For example:
+
+`-E name=${NAME}`
+
+==================================
 
 [float]
 === Examples
@@ -43,9 +68,43 @@ and what each configuration looks like after replacement:
 
 [options="header"]
 |==================================
-|Config source	       |Environment setting   |Config after replacement
-|`name: ${NAME}`       |`export NAME=elastic` |`name: elastic`
-|`name: ${NAME}`       |no setting            |`name:`
-|`name: ${NAME:beats}` |no setting            |`name: beats`
-|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
+|Config source	         |Environment setting   |Config after replacement
+|`name: ${NAME}`         |`export NAME=elastic` |`name: elastic`
+|`name: ${NAME}`         |no setting            |`name:`
+|`name: ${NAME:beats}`   |no setting            |`name: beats`
+|`name: ${NAME:beats}`   |`export NAME=elastic` |`name: elastic`
+|`name: ${NAME:?You need to set the NAME environment variable}`  |no setting            | None. Returns an error message that's prepended with the custom text.
+|`name: ${NAME:?You need to set the NAME environment variable}`  |`export NAME=elastic` | `name: elastic`
 |==================================
+
+[float]
+=== Specifying Complex Objects in Environment Variables
+
+You can specify complex objects, such as lists or dictionaries, in environment
+variables by using a JSON-like syntax.
+
+As with JSON, dictionaries and lists are constructed using `{}` and `[]`. But
+unlike JSON, the syntax allows for trailing commas and slightly different string
+quotation rules. Strings can be unquoted, single-quoted, or double-quoted, as a
+convenience for simple settings and to make it easier for you to mix quotation
+usage in the shell. Arrays at the top-level do not require brackets (`[]`).
+
+For example, the following environment variable is set to a list:
+
+[source,yaml]
+-------------------------------------------------------------------------------
+ES_HOSTS="10.45.3.2:9220,10.45.3.1:9230"
+-------------------------------------------------------------------------------
+
+You can reference this variable in the config file:
+
+[source,yaml]
+-------------------------------------------------------------------------------
+output.elasticsearch:
+  hosts: '${ES_HOSTS}'
+-------------------------------------------------------------------------------
+
+When the Beat loads the config file, it resolves the environment variable and
+replaces it with the specified list before reading the `hosts` setting.
+
+NOTE: Do not use double-quotes (`"`) to wrap regular expressions, or the backslash (`\`) will be interpreted as an escape character.

libbeat/docs/yaml.asciidoc

@@ -6,7 +6,10 @@
 //// Use the appropriate variables defined in the index.asciidoc file to
 //// resolve Beat names: beatname_uc and beatname_lc.
 //// Use the following include to pull this content into a doc file:
+//// :standalone:
 //// include::../../libbeat/docs/yaml.asciidoc[]
+//// Specify :standalone: when this file is pulled into and index. When
+//// the file is embedded in another file, do no specify :standalone:
 //////////////////////////////////////////////////////////////////////////
 
 ifdef::standalone[]
@@ -40,7 +43,7 @@ simply uncomment the line and change the values.
 
 You can test your configuration file to verify that the structure is valid.
 Simply change to the directory where the binary is installed, and run
-the Beat in the foreground with the `-configtest` flag specified. For example: 
+the Beat in the foreground with the `-configtest` flag specified. For example:
 
 ifdef::allplatforms[]
 
@@ -65,7 +68,7 @@ You'll see a message if the Beat finds an error in the file.
 [float]
 === Wrap Regular Expressions in Single Quotation Marks
 
-If you need to specify a regular expression in a YAML file, it's a good idea to wrap the regular expression in single quotation marks to work around YAML's tricky rules for string escaping. 
+If you need to specify a regular expression in a YAML file, it's a good idea to wrap the regular expression in single quotation marks to work around YAML's tricky rules for string escaping.
 
 For more information about YAML, see http://yaml.org/.
 
@@ -74,9 +77,9 @@ For more information about YAML, see http://yaml.org/.
 === Wrap Paths in Single Quotation Marks
 
 Windows paths in particular sometimes contain spaces or characters, such as drive
-letters or triple dots, that may be misinterpreted by the YAML parser. 
+letters or triple dots, that may be misinterpreted by the YAML parser.
 
-To avoid this problem, it's a good idea to wrap paths in single quotation marks. 
+To avoid this problem, it's a good idea to wrap paths in single quotation marks.
 
 [float]
 [[avoid-leading-zeros]]
@@ -85,7 +88,7 @@ To avoid this problem, it's a good idea to wrap paths in single quotation marks.
 If you use a leading zero (for example, `09`) in a numeric field without
 wrapping the value in single quotation marks, the value may be interpreted
 incorrectly by the YAML parser. If the value is a valid octal, it's converted
-to an integer. If not, it's converted to a float. 
+to an integer. If not, it's converted to a float.
 
 To prevent unwanted type conversions, avoid using leading zeros in field values,
 or wrap the values in single quotation marks.

metricbeat/docs/index.asciidoc

@@ -41,6 +41,7 @@ include::../../libbeat/docs/shared-config-ingest.asciidoc[]
 
 include::./configuring-logstash.asciidoc[]
 
+:standalone:
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 :standalone:

packetbeat/docs/index.asciidoc

@@ -51,6 +51,9 @@ include::./configuring-logstash.asciidoc[]
 
 include::./flows.asciidoc[]
 
+:standalone:
+include::../../libbeat/docs/shared-env-vars.asciidoc[]
+
 include::./thrift.asciidoc[]
 
 :standalone:

winlogbeat/docs/index.asciidoc

@@ -34,6 +34,7 @@ include::./winlogbeat-filtering.asciidoc[]
 
 include::../../libbeat/docs/shared-config-ingest.asciidoc[]
 
+:standalone:
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
 :standalone: