-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add json_array_parser parser and assign_keys transformer (#30644)
**Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> We are using otel-collector as an infrastructure and receive many types of data from a client. The client's sent data is always a form of json and in one use case the json is a simple headerless jarray and so we need a way to parse it and match headers to each field (something similar to what csv_parser does but also supports types supported in a json format and nested objects) **Link to tracking Issue:** <Issue number if applicable> #30321 **Testing:** <Describe what testing was performed and which tests were added.> * unittests All the tests found in csv_parser were copied and adjusted adding test scenarios for different types (numbers, booleans, null) as well as a test for parsing a nested object as a part of the jarray * End to end tests Used generated traffic on a running otel collector thats using the parser and verified the data is as expected in the end table **Documentation:** <Describe the documentation added.> * [json_array_parser.md](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/beacea489ff4ae61c0bac4f477c04748944c9054/pkg/stanza/docs/operators/json_array_parser.md) * [assign_keys.md](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/beacea489ff4ae61c0bac4f477c04748944c9054/pkg/stanza/docs/operators/assign_keys.md) --------- Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
- Loading branch information
1 parent
c89b81a
commit aee7b70
Showing
55 changed files
with
1,229 additions
and
0 deletions.
There are no files selected for viewing
18 changes: 18 additions & 0 deletions
18
.chloggen/add_jarray_parser_and_assign_keys_transformer.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' | ||
| change_type: enhancement | ||
|
|
||
| # The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver) | ||
| component: pkg/stanza | ||
|
|
||
| # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). | ||
| note: Add a json array parser operator and an assign keys transformer. | ||
|
|
||
| # One or more tracking issues related to the change | ||
| issues: [30321] | ||
|
|
||
| # (Optional) One or more lines of additional information to render under the primary note. | ||
| # These lines will be padded with 2 spaces and then inserted directly into the document. | ||
| # Use pipe (|) for multiline entries. | ||
| subtext: | | ||
| Json array parser opreator can be used to parse a json array string input into a list of objects. | | ||
| Assign keys transformer can be used to assigns keys from the configuration to an input list |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| ## `assign_keys` operator | ||
|
|
||
| The `assign_keys` assigns keys from the configuration to an input list. the output is a map containing these key-value pairs | ||
|
|
||
| ### Configuration Fields | ||
|
|
||
| | Field | Default | Description | | ||
| | --- | --- | --- | | ||
| | `id` | `assign_keys` | A unique identifier for the operator. | | ||
| | `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | | ||
| | `field` | required | The [field](../types/field.md) to assign keys to. | | ||
| | `keys` | required | The list of strings to be used as the keys to the input list's values. Its length is expected to be equal to the length of the values list from field. In case there is a mismatch, an error will result. | | ||
| | `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](../types/on_error.md). | | ||
| | `if` | | An [expression](../types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | | ||
|
|
||
| ### Example Configurations: | ||
|
|
||
| <hr> | ||
| Assign keys to a list in body | ||
| <br> | ||
| <br> | ||
|
|
||
| ```yaml | ||
| - type: assign_keys | ||
| field: body | ||
| keys: ["foo", "bar", "charlie", "foxtrot"] | ||
| ``` | ||
| <table> | ||
| <tr><td> Input Entry </td> <td> Output Entry </td></tr> | ||
| <tr> | ||
| <td> | ||
| ```json | ||
| { | ||
| "body": [1, "debug", "Debug Message", true] | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| <td> | ||
|
|
||
| ```json | ||
| { | ||
| "body": { | ||
| "foo": 1, | ||
| "bar": "debug", | ||
| "charlie": "Debug Message", | ||
| "foxtrot": true, | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| </tr> | ||
| </table> | ||
| <hr> | ||
| Assign keys to a list in an attributes field | ||
| <br> | ||
| <br> | ||
|
|
||
| ```yaml | ||
| - type: assign_keys | ||
| field: attributes.input | ||
| keys: ["foo", "bar"] | ||
| ``` | ||
| <table> | ||
| <tr><td> Input Entry </td> <td> Output Entry </td></tr> | ||
| <tr> | ||
| <td> | ||
| ```json | ||
| { | ||
| "attributes": { | ||
| "input": [1, "debug"], | ||
| "field2": "unchanged", | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| <td> | ||
|
|
||
| ```json | ||
| { | ||
| "attributes": { | ||
| "input": { | ||
| "foo": 1, | ||
| "bar": "debug", | ||
| }, | ||
| "field2": "unchanged", | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| </tr> | ||
| </table> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| ## `json_array_parser` operator | ||
|
|
||
| The `json_array_parser` operator parses the string-type field selected by `parse_from` assumed to be of a json array format into a list. | ||
| A JArray string (or a json array string) is a string that represents a JSON array. A JSON array is a type of data structure that is used to store data in a structured way. It consists of an ordered list of values that can be either strings, numbers, objects, or even other arrays. | ||
| #### Examples: | ||
| a simple json array string with strictly strings in it: | ||
| ``` | ||
| "[\"Foo\", \"Bar\", \"Charlie\"]" | ||
| ``` | ||
|
|
||
| json array after parsing: | ||
| ```json | ||
| ["Foo", "Bar", "Charlie"] | ||
| ``` | ||
|
|
||
| a more complex json array string with different types in it without nested objects: | ||
| ``` | ||
| "[\"Hello\", 42, true, null]" | ||
| ``` | ||
|
|
||
| json array after parsing: | ||
| ```json | ||
| ["Hello", 42, true, null] | ||
| ``` | ||
|
|
||
| a more complex json array string with different types in it with nested objects: | ||
| ``` | ||
| "[\"Hello\", 42, {\"name\": \"Alice\", \"age\": 25}, [1, 2, 3], true, null]" | ||
| ``` | ||
|
|
||
| json array after parsing: | ||
| ```json | ||
| ["Hello", 42, {"name": "Alice", "age": 25}, [1, 2, 3], true, null] | ||
| ``` | ||
|
|
||
| Notice that for this example, the current parser will parse every nested object as a string and so the result is actually this - | ||
| ```json | ||
| ["Hello", 42, "{\"name\": \"Alice\", \"age\": 25}", "[1, 2, 3]", true, null] | ||
| ``` | ||
|
|
||
| More information on json arrays can be found [here](https://json-schema.org/understanding-json-schema/reference/array) | ||
|
|
||
|
|
||
| ### Configuration Fields | ||
|
|
||
| | Field | Default | Description | | ||
| |--------------------|------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `id` | `json_array_parser` | A unique identifier for the operator. | | ||
| | `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | | ||
| | `parse_from` | `body` | The [field](../types/field.md) from which the value will be parsed. | | ||
| | `parse_to` | required. can be one of `body` or a nested field inside `body`, `attributes` or `resource` (ie `attributes.parsed`) | The [field](../types/field.md) to which the value will be parsed. | | ||
| | `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](../types/on_error.md). | | ||
| | `timestamp` | `nil` | An optional [timestamp](../types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator. | | ||
| | `severity` | `nil` | An optional [severity](../types/severity.md) block which will parse a severity field before passing the entry to the output operator. | | ||
|
|
||
| ### Embedded Operations | ||
|
|
||
| The `json_array_parser` can be configured to embed certain operations such as timestamp and severity parsing. For more information, see [complex parsers](../types/parsers.md#complex-parsers). | ||
|
|
||
| ### Example Configurations | ||
|
|
||
| #### Parse the field `body` with a json array parser into an attributes field | ||
|
|
||
| Configuration: | ||
|
|
||
| ```yaml | ||
| - type: json_array_parser | ||
| parse_from: body | ||
| parse_to: attributes.output | ||
| ``` | ||
| <table> | ||
| <tr><td> Input Entry </td> <td> Output Entry </td></tr> | ||
| <tr> | ||
| <td> | ||
| ```json | ||
| { | ||
| "body": "[1,\"debug\",\"Debug Message\", true]" | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| <td> | ||
|
|
||
| ```json | ||
| { | ||
| "attributes": { | ||
| "output": [1, "debug", "Debug Message", true] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| </tr> | ||
| </table> | ||
|
|
||
| #### Parse the field `body` with a json array parser | ||
|
|
||
| Configuration: | ||
|
|
||
| ```yaml | ||
| - type: json_array_parser | ||
| parse_to: body | ||
| ``` | ||
| <table> | ||
| <tr><td> Input Entry </td> <td> Output Entry </td></tr> | ||
| <tr> | ||
| <td> | ||
| ```json | ||
| { | ||
| "body": "[1,\"debug\",\"Debug Message\", true]" | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| <td> | ||
|
|
||
| ```json | ||
| { | ||
| "body": [1, "debug", "Debug Message", true] | ||
| } | ||
| ``` | ||
|
|
||
| </td> | ||
| </tr> | ||
| </table> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.