From b2b7feb2e76b8210b28c1df2de00c39b04d2bb27 Mon Sep 17 00:00:00 2001 From: Spencer Gilbert Date: Tue, 11 Oct 2022 10:24:39 -0400 Subject: [PATCH 1/5] docs: Update template and field path syntax --- .../configuration/field-path-notation.md | 36 ++++++----- .../configuration/template-syntax.md | 63 ++++++++----------- 2 files changed, 46 insertions(+), 53 deletions(-) diff --git a/website/content/en/docs/reference/configuration/field-path-notation.md b/website/content/en/docs/reference/configuration/field-path-notation.md index 6c3b6d8ad6666..310ee0bfdcc73 100644 --- a/website/content/en/docs/reference/configuration/field-path-notation.md +++ b/website/content/en/docs/reference/configuration/field-path-notation.md @@ -4,7 +4,9 @@ short: Field paths weight: 7 --- -Throughout Vector's configuration you'll notice that certain options take field paths as values. In order to reference nested or array values, you can use Vector's field path notation. This notation is not anything special, it simply uses `.` and `[]` to access nested and array values, respectively. +Throughout Vector's configuration you'll notice that certain options take field paths as values. In order to reference +nested or array values, you can use Vector's field path notation. This notation is not anything special, it simply uses +`.` and `[]` to access nested and array values, respectively. ## Example @@ -17,7 +19,7 @@ Let's take this log event: "message": "Hello world", "field.with.dot": "value", "ec2": { - "instance_id": "abcd1234", + "instance-id": "i-096fba6d03d36d262", "tags": ["tag1: value1", "tag2: value1"] } } @@ -25,11 +27,11 @@ Let's take this log event: We can access the values like so: -Path | Value -:----|:----- -`"application_id"` | The root-level `application_id` field -`"ec2.instance_id"` | The child `instance_id` field -`"ec2.tags[0]"` | The first value in the child `tags` array +| Path | Value | +|:---------------------|:------------------------------------------| +| `.application_id` | The root-level `application_id` field | +| `.ec2."instance-id"` | The child `instance-id` field | +| `.ec2.tags[0]` | The first value in the child `tags` array | ## Syntax @@ -37,38 +39,38 @@ Path | Value Root-level values can be accessed by supplying the name of the field, as shown in the example above: -```toml -field_name +```coffee +.field_name ``` ### Nested values Nested values can be accessed by separating ancestor fields using the `.` character: -```toml -grandparent.parent.child +```coffee +.grandparent.parent.child ``` ### Array Values Array values can be accessed using `[]` syntax. This accesses the first value since it has an index of 0: -```toml -field_name[0] +```coffee +.field_name[0] ``` This accesses the first value of the nested child field: -```toml -parent.child[0] +```coffee +.parent.child[0] ``` ### Escaping The special characters `.`, `[`, and `]` can be escaped with a `\`: -```toml -field\.with\.dots +```coffee +."field.with.dots" ``` The above name is treated literally. diff --git a/website/content/en/docs/reference/configuration/template-syntax.md b/website/content/en/docs/reference/configuration/template-syntax.md index faf802d7b6e0b..293f96746fb7b 100644 --- a/website/content/en/docs/reference/configuration/template-syntax.md +++ b/website/content/en/docs/reference/configuration/template-syntax.md @@ -4,11 +4,13 @@ weight: 6 aliases: ["/docs/reference/templates", "/docs/reference/configuration/templates"] --- -Vector supports a template syntax for some configuration options. This allows for dynamic values derived from event data. Options that support this syntax will be clearly documented as such in the option description. +Vector supports a template syntax for some configuration options. This allows for dynamic values derived from event +data. Any option that supports this syntax will be clearly documented as such in the description. ## Example -For example, let's partition data on AWS S3 by application_id and date. We can accomplish this with the key_prefix option in the aws_s3 sink: +Let's partition data on AWS S3 by "application_id" and "date". We can accomplish this with the `key_prefix` option in +the `aws_s3` sink: ```toml [sinks.backup] @@ -17,7 +19,8 @@ For example, let's partition data on AWS S3 by application_id and date. We can a key_prefix = "application_id={{ application_id }}/date=%F/" ``` -Notice that Vector allows direct field references as well as strftime specifiers. If we were to run the following log event through Vector: +Notice that Vector allows direct field references as well as "strftime" specifiers. If we were to run the following log +event through Vector: ```json { @@ -27,39 +30,42 @@ Notice that Vector allows direct field references as well as strftime specifiers } ``` -The value of the `key_prefix` option would equal: +The value of the above `key_prefix` option would equal: ```raw application_id=1/date=2020-02-14 ``` -Because the [`aws_s3`][aws_s3] sink batches data, each event would be grouped by its produced value. This effectively enables dynamic partitioning, something fundamental to storing log data in filesystems. +Because the [`aws_s3`][aws_s3] sink batches data, each event would be grouped by its produced value. This effectively +enables dynamic partitioning, something fundamental to storing log data in filesystems. ## Syntax ### Event fields -Individual [log event][log] fields can be accessed using `{{ }}` syntax: +Individual [log event][log] fields can be accessed using `{{ ... }}` to wrap a VRL [path expression][path_expressions]: ```toml -option = "{{ field_path_notation }}" +option = "{{ .parent.child }}" ``` -Vector's [field notation][fields] uses `.` to target nested fields and `[]` to target array values. - -### strftime specifiers +### Strftime specifiers In addition to directly accessing fields, Vector offers a shortcut for injecting [strftime specifiers][strftime]: ```toml -options = "year=%Y/month=%m/day=%d/" +option = "year=%Y/month=%m/day=%d/" ``` -The value is derived from the [`timestamp` field][timestamp] and the name of this field can be changed via the [global `timestamp_key` option][timestamp_key]. +{{< info >}} +The value is derived from the [`timestamp` field](/docs/about/under-the-hood/architecture/data-model/log/#timestamps) +and the name of this field can be changed via the [global `timestamp_key` option](/docs/reference/configuration/global-options/#log_schema.timestamp_key). +{{< /info >}} ### Escaping -You can escape this syntax by prefixing the character with a `\`. For example, you can escape the event field syntax like this: +You can escape this syntax by prefixing the character with a `\`. For example, you can escape the event field syntax +like this: ```toml option = "\{{ field_name }}" @@ -68,28 +74,25 @@ option = "\{{ field_name }}" And [strftime] specified like so: ```toml -options = "year=\%Y/month=\%m/day=\%d/" +option = "year=\%Y/month=\%m/day=\%d/" ``` Each of the values above would be treated literally. ## How it works -### Array Values +### Accessing fields -Array values can be accessed using Vector's [field notation syntax][paths]: - -```toml -option = "{{ parent.child[0] }}" -``` +You can find additional examples for accessing fields in the +[path expression reference](docs/reference/vrl/expressions/#path-examples) documentation. ### Fallback values -Vector doesn't currently support fallback values. [Issue 1692][1692] is open to add this functionality. In the interim, you can use the [`remap` transform][remap] to set a default value: +Vector doesn't currently support fallback values, [issue 1692][1692] is open to add this functionality. In the interim, +you can use the [`remap` transform][remap] to set a default value: ```toml [transforms.set_defaults] - # REQUIRED type = "remap" inputs = ["my-source-id"] source = ''' @@ -102,23 +105,11 @@ Vector doesn't currently support fallback values. [Issue 1692][1692] is open to ### Missing fields If a field is missing, an error is logged and Vector drops the event. The `component_errors_total` internal -metric is incremented with `error_type` = `template_failed`. - -### Nested fields - -Nested values can be accessed using Vector's [field notation syntax][paths]: - -```toml -option = "{{ parent.child[0] }}" -``` - +metric is incremented with a `error_type` tag of `template_failed`. [1692]: https://github.com/vectordotdev/vector/issues/1692 [aws_s3]: /docs/reference/configuration/sinks/aws_s3 -[fields]: /docs/reference/configuration/field-path-notation [log]: /docs/about/under-the-hood/architecture/data-model/log -[paths]: /docs/reference/configuration/field-path-notation +[path_expressions]: /docs/reference/vrl/expressions/#path [remap]: /docs/reference/configuration/transforms/remap [strftime]: https://docs.rs/chrono/0.4.19/chrono/format/strftime/index.html#specifiers -[timestamp]: /docs/about/under-the-hood/architecture/data-model/log/#timestamps -[timestamp_key]: /docs/reference/configuration/global-options/#log_schema.timestamp_key From cd2276dc7a1a4a26b9698d745c09485e0e5def17 Mon Sep 17 00:00:00 2001 From: Spencer Gilbert Date: Fri, 14 Oct 2022 15:00:40 -0400 Subject: [PATCH 2/5] +netlify redirect path-fields to vrl/expressions/#path Signed-off-by: Spencer Gilbert --- netlify.toml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/netlify.toml b/netlify.toml index 809b7cc94e82c..dad74bbc15e0a 100644 --- a/netlify.toml +++ b/netlify.toml @@ -88,6 +88,12 @@ to = "https://www.datadoghq.com/product/observability-pipelines" status = 301 force = true +# Removed pages redirects +[[redirects]] +from = "/docs/reference/configuration/field-path-notation/" +to = "/docs/reference/vrl/expressions/#path" +force = true + # CORS headers for /index.json [[headers]] for = "/index.json" From b29473ff1fe166020ef0936db47f4c1a810f16fb Mon Sep 17 00:00:00 2001 From: Spencer Gilbert Date: Fri, 14 Oct 2022 15:02:48 -0400 Subject: [PATCH 3/5] +delete old field-path page Signed-off-by: Spencer Gilbert --- .../configuration/field-path-notation.md | 78 ------------------- 1 file changed, 78 deletions(-) delete mode 100644 website/content/en/docs/reference/configuration/field-path-notation.md diff --git a/website/content/en/docs/reference/configuration/field-path-notation.md b/website/content/en/docs/reference/configuration/field-path-notation.md deleted file mode 100644 index 310ee0bfdcc73..0000000000000 --- a/website/content/en/docs/reference/configuration/field-path-notation.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: Field path notation reference -short: Field paths -weight: 7 ---- - -Throughout Vector's configuration you'll notice that certain options take field paths as values. In order to reference -nested or array values, you can use Vector's field path notation. This notation is not anything special, it simply uses -`.` and `[]` to access nested and array values, respectively. - -## Example - -Let's take this log event: - -```json -{ - "timestamp": "2020-02-14T01:22:23.223Z", - "application_id": 1, - "message": "Hello world", - "field.with.dot": "value", - "ec2": { - "instance-id": "i-096fba6d03d36d262", - "tags": ["tag1: value1", "tag2: value1"] - } -} -``` - -We can access the values like so: - -| Path | Value | -|:---------------------|:------------------------------------------| -| `.application_id` | The root-level `application_id` field | -| `.ec2."instance-id"` | The child `instance-id` field | -| `.ec2.tags[0]` | The first value in the child `tags` array | - -## Syntax - -### Root-level values - -Root-level values can be accessed by supplying the name of the field, as shown in the example above: - -```coffee -.field_name -``` - -### Nested values - -Nested values can be accessed by separating ancestor fields using the `.` character: - -```coffee -.grandparent.parent.child -``` - -### Array Values - -Array values can be accessed using `[]` syntax. This accesses the first value since it has an index of 0: - -```coffee -.field_name[0] -``` - -This accesses the first value of the nested child field: - -```coffee -.parent.child[0] -``` - -### Escaping - -The special characters `.`, `[`, and `]` can be escaped with a `\`: - -```coffee -."field.with.dots" -``` - -The above name is treated literally. - -The `\` character, if used literally, must be escaped with a `\` as well. From 41dd5de4f4b757681aa2bd8cc6e6485a3430fd7f Mon Sep 17 00:00:00 2001 From: Spencer Gilbert Date: Fri, 14 Oct 2022 15:46:13 -0400 Subject: [PATCH 4/5] +fix link Signed-off-by: Spencer Gilbert --- .../en/docs/reference/configuration/template-syntax.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/website/content/en/docs/reference/configuration/template-syntax.md b/website/content/en/docs/reference/configuration/template-syntax.md index 293f96746fb7b..30e6e75a3bcae 100644 --- a/website/content/en/docs/reference/configuration/template-syntax.md +++ b/website/content/en/docs/reference/configuration/template-syntax.md @@ -43,7 +43,7 @@ enables dynamic partitioning, something fundamental to storing log data in files ### Event fields -Individual [log event][log] fields can be accessed using `{{ ... }}` to wrap a VRL [path expression][path_expressions]: +Individual [log event][log] fields can be accessed using `{{ ... }}` to wrap a VRL [path expression][path_expression]: ```toml option = "{{ .parent.child }}" @@ -84,7 +84,7 @@ Each of the values above would be treated literally. ### Accessing fields You can find additional examples for accessing fields in the -[path expression reference](docs/reference/vrl/expressions/#path-examples) documentation. +[path expression reference][path_expression_examples] documentation. ### Fallback values @@ -110,6 +110,7 @@ metric is incremented with a `error_type` tag of `template_failed`. [1692]: https://github.com/vectordotdev/vector/issues/1692 [aws_s3]: /docs/reference/configuration/sinks/aws_s3 [log]: /docs/about/under-the-hood/architecture/data-model/log -[path_expressions]: /docs/reference/vrl/expressions/#path +[path_expression]: /docs/reference/vrl/expressions/#path +[path_expression_examples]: /docs/reference/vrl/expressions/#path-examples [remap]: /docs/reference/configuration/transforms/remap [strftime]: https://docs.rs/chrono/0.4.19/chrono/format/strftime/index.html#specifiers From 45ee38526010a0a59ae9a120506211f4332b3599 Mon Sep 17 00:00:00 2001 From: Spencer Gilbert Date: Fri, 14 Oct 2022 16:04:53 -0400 Subject: [PATCH 5/5] Update website/content/en/docs/reference/configuration/template-syntax.md Co-authored-by: neuronull --- .../content/en/docs/reference/configuration/template-syntax.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/en/docs/reference/configuration/template-syntax.md b/website/content/en/docs/reference/configuration/template-syntax.md index 30e6e75a3bcae..104f7c0aea828 100644 --- a/website/content/en/docs/reference/configuration/template-syntax.md +++ b/website/content/en/docs/reference/configuration/template-syntax.md @@ -105,7 +105,7 @@ you can use the [`remap` transform][remap] to set a default value: ### Missing fields If a field is missing, an error is logged and Vector drops the event. The `component_errors_total` internal -metric is incremented with a `error_type` tag of `template_failed`. +metric is incremented with an `error_type` tag of `template_failed`. [1692]: https://github.com/vectordotdev/vector/issues/1692 [aws_s3]: /docs/reference/configuration/sinks/aws_s3