Skip to content

Commit

Permalink
ingest: document fields that support templating
Browse files Browse the repository at this point in the history 
This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of elastic#33188
  • Loading branch information
@jakelandis
jakelandis committed Oct 16, 2018
1 parent 230ad53 commit 09d345b
Showing 1 changed file with 40 additions and 37 deletions.
77 changes: 40 additions & 37 deletions docs/reference/ingest/ingest-node.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -751,23 +751,23 @@ pipeline basis. Useful to find out which pipelines are used the most or spent th
Appends one or more values to an existing array if the field already exists and it is an array.
Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar.
Creates an array containing the provided values if the field doesn't exist.
Accepts a single value or an array of values.
Accepts a single value or an array of values. Settings that support <<accessing-template-fields,template values>> are annotated with ^T^.

[[append-options]]
.Append Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be appended to
| `value` | yes | - | The value to be appended
| `field`^T^ | yes | - | The field to be appended to
| `value`^T^ | yes | - | The value to be appended
|======

[source,js]
--------------------------------------------------
{
"append": {
"field": "field1",
"value": ["item2", "item3", "item4"]
"field": "tags",
"value": ["production", "{{app}}", "{{owner}}"]
}
}
--------------------------------------------------
Expand All @@ -794,7 +794,7 @@ the field is not a supported format or resultant value exceeds 2^63.
--------------------------------------------------
{
"bytes": {
"field": "foo"
"field": "file.size"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -832,7 +832,7 @@ still be updated with the unconverted field value.
--------------------------------------------------
{
"convert": {
"field" : "foo",
"field" : "url.port",
"type": "integer"
}
}
Expand All @@ -846,7 +846,8 @@ Parses dates from fields, and then uses the date or timestamp as the timestamp f
By default, the date processor adds the parsed date as a new field called `@timestamp`. You can specify a
different field by setting the `target_field` configuration parameter. Multiple date formats are supported
as part of the same date processor definition. They will be used sequentially to attempt parsing the date field,
in the same order they were defined as part of the processor definition.
in the same order they were defined as part of the processor definition. Settings that support
<<accessing-template-fields,template values>> are annotated with ^T^.

[[date-options]]
.Date options
Expand All @@ -856,8 +857,8 @@ in the same order they were defined as part of the processor definition.
| `field` | yes | - | The field to get the date from.
| `target_field` | no | @timestamp | The field that will hold the parsed date.
| `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days.
| `timezone`^T^ | no | UTC | The timezone to use when parsing the date.
| `locale`^T^ | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days.
|======

Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field:
Expand Down Expand Up @@ -895,8 +896,8 @@ the timezone and locale values.
"field" : "initial_date",
"target_field" : "timestamp",
"formats" : ["ISO8601"],
"timezone" : "{{ my_timezone }}",
"locale" : "{{ my_locale }}"
"timezone" : "{{my_timezone}}",
"locale" : "{{my_locale}}"
}
}
]
Expand All @@ -917,7 +918,7 @@ First, this processor fetches the date or timestamp from a field in the document
date formatting can be configured on how the field's value should be parsed into a date. Then this date,
the provided index name prefix and the provided date rounding get formatted into a date math index name expression.
Also here optionally date formatting can be specified on how the date should be formatted into a date math index name
expression.
expression. Settings that support <<accessing-template-fields,template values>> are annotated with ^T^.

An example pipeline that points documents to a monthly index that starts with a `myindex-` prefix based on a
date in the `date1` field:
Expand Down Expand Up @@ -1041,12 +1042,12 @@ understands this to mean `2016-04-01` as is explained in the <<date-math-index-n
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to get the date or timestamp from.
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second).
| `index_name_prefix`^T^ | no | - | A prefix of the index name to be prepended before the printed date.
| `date_rounding`^T^ | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second).
| `date_formats` | no | yyyy-MM-dd'T'HH:mm:ss.SSSZ | An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
| `locale` | no | ENGLISH | The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here.
| `index_name_format`^T^ | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here.
|======

[[dissect-processor]]
Expand Down Expand Up @@ -1365,21 +1366,22 @@ a scalar field to an object field.
=== Fail Processor
Raises an exception. This is useful for when
you expect a pipeline to fail and want to relay a specific message
to the requester.
to the requester. Settings that support <<accessing-template-fields,template values>> are annotated with ^T^.

[[fail-options]]
.Fail Options
[options="header"]
|======
| Name | Required | Default | Description
| `message` | yes | - | The error message of the `FailException` thrown by the processor
| `message`^T^ | yes | - | The error message thrown by the processor
|======

[source,js]
--------------------------------------------------
{
"fail": {
"message": "an error message"
"if" : "ctx.tags.contains('production') != true",
"message": "The production tag is not present, found tags: {{tags}}"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2077,14 +2079,14 @@ Converts a string to its lowercase equivalent.

[[remove-processor]]
=== Remove Processor
Removes existing fields. If one field doesn't exist, an exception will be thrown.
Removes existing fields. If one field doesn't exist, an exception will be thrown. Settings that support <<accessing-template-fields,template values>> are annotated with ^T^.

[[remove-options]]
.Remove Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | Fields to be removed
| `field`^T^ | yes | - | Fields to be removed
| `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document
|======

Expand All @@ -2094,7 +2096,7 @@ Here is an example to remove a single field:
--------------------------------------------------
{
"remove": {
"field": "foo"
"field": "user_agent"
}
}
--------------------------------------------------
Expand All @@ -2106,7 +2108,7 @@ To remove multiple fields, you can use the following query:
--------------------------------------------------
{
"remove": {
"field": ["foo", "bar"]
"field": ["user_agent", "url"]
}
}
--------------------------------------------------
Expand All @@ -2115,23 +2117,24 @@ To remove multiple fields, you can use the following query:
[[rename-processor]]
=== Rename Processor
Renames an existing field. If the field doesn't exist or the new name is already used, an exception will be thrown.
Settings that support <<accessing-template-fields,template values>> are annotated with ^T^.

[[rename-options]]
.Rename Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be renamed
| `target_field` | yes | - | The new name of the field
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
| Name | Required | Default | Description
| `field`^T^ | yes | - | The field to be renamed
| `target_field`^T^ | yes | - | The new name of the field
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
|======

[source,js]
--------------------------------------------------
{
"rename": {
"field": "foo",
"target_field": "foobar"
"field": "provider",
"target_field": "cloud.provider"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2243,24 +2246,24 @@ and depending on the progress made, indexed into different indices.
[[set-processor]]
=== Set Processor
Sets one field and associates it with the specified value. If the field already exists,
its value will be replaced with the provided one.
its value will be replaced with the provided one. Settings that support <<accessing-template-fields,template values>> are annotated with ^T^.

[[set-options]]
.Set Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update
| `value` | yes | - | The value to be set for the field
| `override`| no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
| Name | Required | Default | Description
| `field`^T^ | yes | - | The field to insert, upsert, or update
| `value`^T^ | yes | - | The value to be set for the field
| `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
|======

[source,js]
--------------------------------------------------
{
"set": {
"field": "field1",
"value": 582.1
"field": "host.os.name",
"value": "{{os}}"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2313,7 +2316,7 @@ Throws an error when the field is not an array.
--------------------------------------------------
{
"sort": {
"field": "field_to_sort",
"field": "array_field_to_sort",
"order": "desc"
}
}
Expand Down

0 comments on commit 09d345b

Please sign in to comment.