From 3a59d5e8736e6dfd73a59a1c052c68fdc1430712 Mon Sep 17 00:00:00 2001 From: Jake Landis Date: Thu, 1 Nov 2018 11:08:04 -0500 Subject: [PATCH] ingest: add common options to each processor's documentation (#35091) * adds `if`, `on_failure`, `tag`, and `ignore_failure` to table for each processor part of #33188 * added ingore_failure * fix whitespace noise --- .../ingest-node-common-processor.asciidoc | 5 +++ docs/reference/ingest/ingest-node.asciidoc | 35 +++++++++++++++++-- 2 files changed, 38 insertions(+), 2 deletions(-) create mode 100644 docs/reference/ingest/ingest-node-common-processor.asciidoc diff --git a/docs/reference/ingest/ingest-node-common-processor.asciidoc b/docs/reference/ingest/ingest-node-common-processor.asciidoc new file mode 100644 index 000000000000..dcf8b63630b4 --- /dev/null +++ b/docs/reference/ingest/ingest-node-common-processor.asciidoc @@ -0,0 +1,5 @@ +| `if` | no | - | Conditionally execute this processor. +| `on_failure` | no | - | Handle failures for this processor. See <>. +| `ignore_failure` | no | `false` | Ignore failures for this processor. See <>. +| `tag` | no | - | An identifier for this processor. Useful for debugging and metrics. +// TODO: See <>. <-- for the if description once PR 35044 is merged \ No newline at end of file diff --git a/docs/reference/ingest/ingest-node.asciidoc b/docs/reference/ingest/ingest-node.asciidoc index 2a26b0e5d427..9ba5191b3cf3 100644 --- a/docs/reference/ingest/ingest-node.asciidoc +++ b/docs/reference/ingest/ingest-node.asciidoc @@ -778,6 +778,7 @@ Accepts a single value or an array of values. | Name | Required | Default | Description | `field` | yes | - | The field to be appended to. Supports <>. | `value` | yes | - | The value to be appended. Supports <>. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -806,6 +807,7 @@ the field is not a supported format or resultant value exceeds 2^63. | `field` | yes | - | The field to convert | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -844,6 +846,7 @@ still be updated with the unconverted field value. | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `type` | yes | - | The type to convert the existing value to | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -876,6 +879,7 @@ in the same order they were defined as part of the processor definition. | `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. | `timezone` | no | UTC | The timezone to use when parsing the date. Supports <>. | `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. Supports <>. +include::ingest-node-common-processor.asciidoc[] |====== Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field: @@ -1065,6 +1069,7 @@ understands this to mean `2016-04-01` as is explained in the <>. +include::ingest-node-common-processor.asciidoc[] |====== [[dissect-processor]] @@ -1134,7 +1139,7 @@ See <> for more information. | `pattern` | yes | - | The pattern to apply to the field | `append_separator`| no | "" (empty string) | The character(s) that separate the appended fields. | `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document -| ` +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1162,7 +1167,6 @@ modifiers. | `+` with `/n` | Append with order | left and right | `%{+keyname/2} %{+keyname/1}` | Appends two or more fields together in the order specified | <> | `?` | Named skip key | left | `%{?ignoreme}` | Skips the matched value in the output. Same behavior as `%{}`| <> | `*` and `&` | Reference keys | left | `%{*r1} %{&r1}` | Sets the output key as value of `*` and output value of `&` | <> -| ` |====== [[dissect-modifier-skip-right-padding]] @@ -1265,6 +1269,14 @@ Reference key modifier example Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition. +[[drop-options]] +.Drop Options +[options="header"] +|====== +| Name | Required | Default | Description +include::ingest-node-common-processor.asciidoc[] +|====== + [source,js] -------------------------------------------------- { @@ -1289,6 +1301,7 @@ Otherwise these <> can't be accessed by any | Name | Required | Default | Description | `field` | yes | - | The field to expand into an object field | `path` | no | - | The field that contains the field to expand. Only required if the field to expand is part another object field, because the `field` option can only understand leaf fields. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1406,6 +1419,7 @@ to the requester. |====== | Name | Required | Default | Description | `message` | yes | - | The error message thrown by the processor. Supports <>. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1452,6 +1466,7 @@ then it aborts the execution and leaves the array unmodified. | `field` | yes | - | The array field | `processor` | yes | - | The processor to execute against each field | `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== Assume the following document: @@ -1645,6 +1660,7 @@ Grok expression. | `pattern_definitions` | no | - | A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition. | `trace_match` | no | false | when true, `_ingest._grok_match_index` will be inserted into your matched document's metadata with the index into the pattern found in `patterns` that matched. | `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== Here is an example of using the provided patterns to extract out and name structured fields from a string field in @@ -1919,6 +1935,7 @@ If the field is not a string, the processor will throw an exception. | `replacement` | yes | - | The string to replace the matching patterns with | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1946,6 +1963,7 @@ Throws an error when the field is not an array. | `field` | yes | - | The field to be separated | `separator` | yes | - | The separator character | `target_field` | no | `field` | The field to assign the joined value to, by default `field` is updated in-place +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -1971,6 +1989,7 @@ Converts a JSON string into a structured JSON object. | `field` | yes | - | The field to be parsed | `target_field` | no | `field` | The field to insert the converted structured object into | `add_to_root` | no | false | Flag that forces the serialized json to be injected into the top level of the document. `target_field` must not be set when this option is chosen. +include::ingest-node-common-processor.asciidoc[] |====== All JSON-supported types will be parsed (null, boolean, number, array, object, string). @@ -2082,6 +2101,7 @@ For example, if you have a log message which contains `ip=1.2.3.4 error=REFUSED` | `trim_key` | no | `null` | String of characters to trim from extracted keys | `trim_value` | no | `null` | String of characters to trim from extracted values | `strip_brackets` | no | `false` | If `true` strip brackets `()`, `<>`, `[]` as well as quotes `'` and `"` from extracted values +include::ingest-node-common-processor.asciidoc[] |====== @@ -2097,6 +2117,7 @@ Converts a string to its lowercase equivalent. | `field` | yes | - | The field to make lowercase | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2119,6 +2140,7 @@ Executes another pipeline. |====== | Name | Required | Default | Description | `name` | yes | - | The name of the pipeline to execute +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2234,6 +2256,7 @@ Removes existing fields. If one field doesn't exist, an exception will be thrown | Name | Required | Default | Description | `field` | yes | - | Fields to be removed. Supports <>. | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== Here is an example to remove a single field: @@ -2272,6 +2295,7 @@ Renames an existing field. If the field doesn't exist or the new name is already | `field` | yes | - | The field to be renamed. Supports <>. | `target_field` | yes | - | The new name of the field. Supports <>. | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2305,6 +2329,7 @@ caching see <>. | `id` | no | - | The stored script id to refer to | `source` | no | - | An inline script to be executed | `params` | no | - | Script Parameters +include::ingest-node-common-processor.asciidoc[] |====== One of `id` or `source` options must be provided in order to properly reference a script to execute. @@ -2401,6 +2426,7 @@ its value will be replaced with the provided one. | `field` | yes | - | The field to insert, upsert, or update. Supports <>. | `value` | yes | - | The value to be set for the field. Supports <>. | `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched. +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2427,6 +2453,7 @@ Splits a field into an array using a separator character. Only works on string f | `separator` | yes | - | A regex which matches the separator, eg `,` or `\s+` | `target_field` | no | `field` | The field to assign the split value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2455,6 +2482,7 @@ Throws an error when the field is not an array. | `field` | yes | - | The field to be sorted | `order` | no | `"asc"` | The sort order to use. Accepts `"asc"` or `"desc"`. | `target_field` | no | `field` | The field to assign the sorted value to, by default `field` is updated in-place +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2482,6 +2510,7 @@ NOTE: This only works on leading and trailing whitespace. | `field` | yes | - | The string-valued field to trim whitespace from | `target_field` | no | `field` | The field to assign the trimmed value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2506,6 +2535,7 @@ Converts a string to its uppercase equivalent. | `field` | yes | - | The field to make uppercase | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js] @@ -2530,6 +2560,7 @@ URL-decodes a string | `field` | yes | - | The field to decode | `target_field` | no | `field` | The field to assign the converted value to, by default `field` is updated in-place | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document +include::ingest-node-common-processor.asciidoc[] |====== [source,js]