Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Update template and field path syntax #14849

Merged
merged 5 commits into from
Oct 14, 2022
Merged

docs: Update template and field path syntax #14849

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b2b7feb
docs: Update template and field path syntax
spencergilbert Oct 11, 2022
cd2276d
+netlify redirect path-fields to vrl/expressions/#path
spencergilbert Oct 14, 2022
b29473f
+delete old field-path page
spencergilbert Oct 14, 2022
41dd5de
+fix link
spencergilbert Oct 14, 2022
45ee385
Update website/content/en/docs/reference/configuration/template-synta…
spencergilbert Oct 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions netlify.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,12 @@ to = "https://www.datadoghq.com/product/observability-pipelines"
status = 301
force = true

# Removed pages redirects
[[redirects]]
from = "/docs/reference/configuration/field-path-notation/"
to = "/docs/reference/vrl/expressions/#path"
force = true

# CORS headers for /index.json
[[headers]]
for = "/index.json"
Expand Down
76 changes: 0 additions & 76 deletions website/content/en/docs/reference/configuration/field-path-notation.md
View file
Open in desktop

This file was deleted.

64 changes: 28 additions & 36 deletions website/content/en/docs/reference/configuration/template-syntax.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ weight: 6
aliases: ["/docs/reference/templates", "/docs/reference/configuration/templates"]
---

Vector supports a template syntax for some configuration options. This allows for dynamic values derived from event data. Options that support this syntax will be clearly documented as such in the option description.
Vector supports a template syntax for some configuration options. This allows for dynamic values derived from event
data. Any option that supports this syntax will be clearly documented as such in the description.

## Example

For example, let's partition data on AWS S3 by application_id and date. We can accomplish this with the key_prefix option in the aws_s3 sink:
Let's partition data on AWS S3 by "application_id" and "date". We can accomplish this with the `key_prefix` option in
the `aws_s3` sink:

```toml
[sinks.backup]
Expand All @@ -17,7 +19,8 @@ For example, let's partition data on AWS S3 by application_id and date. We can a
key_prefix = "application_id={{ application_id }}/date=%F/"
```

Notice that Vector allows direct field references as well as strftime specifiers. If we were to run the following log event through Vector:
Notice that Vector allows direct field references as well as "strftime" specifiers. If we were to run the following log
event through Vector:

```json
{
Expand All @@ -27,39 +30,42 @@ Notice that Vector allows direct field references as well as strftime specifiers
}
```

The value of the `key_prefix` option would equal:
The value of the above `key_prefix` option would equal:

```raw
application_id=1/date=2020-02-14
```

Because the [`aws_s3`][aws_s3] sink batches data, each event would be grouped by its produced value. This effectively enables dynamic partitioning, something fundamental to storing log data in filesystems.
Because the [`aws_s3`][aws_s3] sink batches data, each event would be grouped by its produced value. This effectively
enables dynamic partitioning, something fundamental to storing log data in filesystems.

## Syntax

### Event fields

Individual [log event][log] fields can be accessed using `{{ <field-path-notation> }}` syntax:
Individual [log event][log] fields can be accessed using `{{ ... }}` to wrap a VRL [path expression][path_expression]:

```toml
option = "{{ field_path_notation }}"
option = "{{ .parent.child }}"
```

Vector's [field notation][fields] uses `.` to target nested fields and `[<index>]` to target array values.

### strftime specifiers
### Strftime specifiers

In addition to directly accessing fields, Vector offers a shortcut for injecting [strftime specifiers][strftime]:

```toml
options = "year=%Y/month=%m/day=%d/"
option = "year=%Y/month=%m/day=%d/"
```

The value is derived from the [`timestamp` field][timestamp] and the name of this field can be changed via the [global `timestamp_key` option][timestamp_key].
{{< info >}}
The value is derived from the [`timestamp` field](/docs/about/under-the-hood/architecture/data-model/log/#timestamps)
and the name of this field can be changed via the [global `timestamp_key` option](/docs/reference/configuration/global-options/#log_schema.timestamp_key).
{{< /info >}}

### Escaping

You can escape this syntax by prefixing the character with a `\`. For example, you can escape the event field syntax like this:
You can escape this syntax by prefixing the character with a `\`. For example, you can escape the event field syntax
like this:

```toml
option = "\{{ field_name }}"
Expand All @@ -68,28 +74,25 @@ option = "\{{ field_name }}"
And [strftime] specified like so:

```toml
options = "year=\%Y/month=\%m/day=\%d/"
option = "year=\%Y/month=\%m/day=\%d/"
```

Each of the values above would be treated literally.

## How it works

### Array Values
### Accessing fields

Array values can be accessed using Vector's [field notation syntax][paths]:

```toml
option = "{{ parent.child[0] }}"
```
You can find additional examples for accessing fields in the
[path expression reference][path_expression_examples] documentation.

### Fallback values

Vector doesn't currently support fallback values. [Issue 1692][1692] is open to add this functionality. In the interim, you can use the [`remap` transform][remap] to set a default value:
Vector doesn't currently support fallback values, [issue 1692][1692] is open to add this functionality. In the interim,
you can use the [`remap` transform][remap] to set a default value:

```toml
[transforms.set_defaults]
# REQUIRED
type = "remap"
inputs = ["my-source-id"]
source = '''
Expand All @@ -102,23 +105,12 @@ Vector doesn't currently support fallback values. [Issue 1692][1692] is open to
### Missing fields

If a field is missing, an error is logged and Vector drops the event. The `component_errors_total` internal
metric is incremented with `error_type` = `template_failed`.

### Nested fields

Nested values can be accessed using Vector's [field notation syntax][paths]:

```toml
option = "{{ parent.child[0] }}"
```

metric is incremented with a `error_type` tag of `template_failed`.
spencergilbert marked this conversation as resolved.
Show resolved Hide resolved

[1692]: https://github.com/vectordotdev/vector/issues/1692
[aws_s3]: /docs/reference/configuration/sinks/aws_s3
[fields]: /docs/reference/configuration/field-path-notation
[log]: /docs/about/under-the-hood/architecture/data-model/log
[paths]: /docs/reference/configuration/field-path-notation
[path_expression]: /docs/reference/vrl/expressions/#path
[path_expression_examples]: /docs/reference/vrl/expressions/#path-examples
[remap]: /docs/reference/configuration/transforms/remap
[strftime]: https://docs.rs/chrono/0.4.19/chrono/format/strftime/index.html#specifiers
[timestamp]: /docs/about/under-the-hood/architecture/data-model/log/#timestamps
[timestamp_key]: /docs/reference/configuration/global-options/#log_schema.timestamp_key