Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingest: document fields that support templating #34536

Merged
merged 7 commits into from
Oct 23, 2018
Merged

ingest: document fields that support templating #34536

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
09d345b
ingest: document fields that support templating
jakelandis Oct 16, 2018
f531257
remove superscript in favor of description
jakelandis Oct 19, 2018
2b5ce43
values -> snippets for consitency
jakelandis Oct 19, 2018
ff7e902
Merge branch 'master' into template_doc
jakelandis Oct 20, 2018
2ae3546
Merge branch 'master' into template_doc
jakelandis Oct 21, 2018
b7f8725
Merge branch 'master' into template_doc
jakelandis Oct 22, 2018
fe0c9b1
Merge branch 'master' into template_doc
jakelandis Oct 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 32 additions & 31 deletions docs/reference/ingest/ingest-node.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -758,16 +758,16 @@ Accepts a single value or an array of values.
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be appended to
| `value` | yes | - | The value to be appended
| `field` | yes | - | The field to be appended to. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be appended. Supports <<accessing-template-fields,template snippets>>.
|======

[source,js]
--------------------------------------------------
{
"append": {
"field": "field1",
"value": ["item2", "item3", "item4"]
"field": "tags",
"value": ["production", "{{app}}", "{{owner}}"]
}
}
--------------------------------------------------
Expand All @@ -794,7 +794,7 @@ the field is not a supported format or resultant value exceeds 2^63.
--------------------------------------------------
{
"bytes": {
"field": "foo"
"field": "file.size"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -832,7 +832,7 @@ still be updated with the unconverted field value.
--------------------------------------------------
{
"convert": {
"field" : "foo",
"field" : "url.port",
"type": "integer"
}
}
Expand All @@ -856,8 +856,8 @@ in the same order they were defined as part of the processor definition.
| `field` | yes | - | The field to get the date from.
| `target_field` | no | @timestamp | The field that will hold the parsed date.
| `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days.
| `timezone` | no | UTC | The timezone to use when parsing the date. Supports <<accessing-template-fields,template snippets>>.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. Supports <<accessing-template-fields,template snippets>>.
|======

Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field:
Expand Down Expand Up @@ -895,8 +895,8 @@ the timezone and locale values.
"field" : "initial_date",
"target_field" : "timestamp",
"formats" : ["ISO8601"],
"timezone" : "{{ my_timezone }}",
"locale" : "{{ my_locale }}"
"timezone" : "{{my_timezone}}",
"locale" : "{{my_locale}}"
}
}
]
Expand Down Expand Up @@ -1041,12 +1041,12 @@ understands this to mean `2016-04-01` as is explained in the <<date-math-index-n
|======
Copy link
Contributor Author

@jakelandis jakelandis Oct 16, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

| Name | Required | Default | Description
| `field` | yes | - | The field to get the date or timestamp from.
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second).
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date. Supports <<accessing-template-fields,template snippets>>.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second). Supports <<accessing-template-fields,template snippets>>.
| `date_formats` | no | yyyy-MM-dd'T'HH:mm:ss.SSSZ | An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
| `locale` | no | ENGLISH | The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here. Supports <<accessing-template-fields,template snippets>>.
|======

[[dissect-processor]]
Expand Down Expand Up @@ -1372,14 +1372,15 @@ to the requester.
[options="header"]
|======
| Name | Required | Default | Description
| `message` | yes | - | The error message of the `FailException` thrown by the processor
| `message` | yes | - | The error message thrown by the processor. Supports <<accessing-template-fields,template snippets>>.
|======

[source,js]
--------------------------------------------------
{
"fail": {
"message": "an error message"
"if" : "ctx.tags.contains('production') != true",
"message": "The production tag is not present, found tags: {{tags}}"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2084,7 +2085,7 @@ Removes existing fields. If one field doesn't exist, an exception will be thrown
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | Fields to be removed
| `field` | yes | - | Fields to be removed. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document
|======

Expand All @@ -2094,7 +2095,7 @@ Here is an example to remove a single field:
--------------------------------------------------
{
"remove": {
"field": "foo"
"field": "user_agent"
}
}
--------------------------------------------------
Expand All @@ -2106,7 +2107,7 @@ To remove multiple fields, you can use the following query:
--------------------------------------------------
{
"remove": {
"field": ["foo", "bar"]
"field": ["user_agent", "url"]
}
}
--------------------------------------------------
Expand All @@ -2120,18 +2121,18 @@ Renames an existing field. If the field doesn't exist or the new name is already
.Rename Options
[options="header"]
Copy link
Contributor Author

@jakelandis jakelandis Oct 16, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be renamed
| `target_field` | yes | - | The new name of the field
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
| Name | Required | Default | Description
| `field` | yes | - | The field to be renamed. Supports <<accessing-template-fields,template snippets>>.
| `target_field` | yes | - | The new name of the field. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
|======

[source,js]
--------------------------------------------------
{
"rename": {
"field": "foo",
"target_field": "foobar"
"field": "provider",
"target_field": "cloud.provider"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2249,18 +2250,18 @@ its value will be replaced with the provided one.
.Set Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update
| `value` | yes | - | The value to be set for the field
| `override`| no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
| Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be set for the field. Supports <<accessing-template-fields,template snippets>>.
| `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
|======

[source,js]
--------------------------------------------------
{
"set": {
"field": "field1",
"value": 582.1
"field": "host.os.name",
"value": "{{os}}"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2313,7 +2314,7 @@ Throws an error when the field is not an array.
--------------------------------------------------
{
"sort": {
"field": "field_to_sort",
"field": "array_field_to_sort",
"order": "desc"
}
}
Expand Down