log data in JSON format #21915

soulg24 · 2024-11-29T06:51:02Z

soulg24
Nov 29, 2024

Good afternoon!
I'm working on setting up log transmission to OpenSearch using a JSON log format. My test attempts have been successful, but I need to find a way to configure Vector to send log data in JSON format without sending the entire log in one message.
I've tried specifying the type like this:

[sources.my_source]
type = "file"
include = ["/path/to/your/logs/log.json"]
file_type = "json"
I tried the encoding.codec = "json" option

However, I encountered an error. What is the correct way to do this?

Answered by pront

Dec 4, 2024

Here is a sample config:

sources:
  source0:
    type: file
    include:
      - "/path/to/log.json"

transforms:
  transform0:
    type: remap
    inputs:
      - source0
    source: |
      .message = parse_json!(string!(.message))

sinks:
  console:
    type: console
    inputs: ["transform0"]
    encoding:
      codec: json
      json:
        pretty: true

The magic happens at .message = parse_json!(string!(.message)) it basically takes the bytes from file (line by line) and constructs an object map. And finally it overwrites the .message with the result.

You try this on the VRL playground. See example.

View full answer

pront · 2024-12-02T18:01:30Z

pront
Dec 2, 2024
Maintainer

What does your log.json look like? Is it newline separated JSON strings?

1 reply

soulg24 Dec 4, 2024
Author

Как это log.json выглядит? Это строки JSON, разделенные новой строкой?

Yes, this is a json string.

{"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
{"timestamp": "2024-12-04 10:11:43.037671", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
{"timestamp": "2024-12-04 10:11:43.608739", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}

jszwedko · 2024-12-04T15:22:29Z

jszwedko
Dec 4, 2024
Maintainer

I'm a bit confused here, neither file_type or encoding.codec are options that exist on the file source. Adding decoding options to the file source is being tracked by #9210.

If you want to parse the messages as JSON you will need to use a remap transform with the parse_json function.

4 replies

soulg24 Dec 4, 2024
Author

I'm a bit confused here, neither file_type or encoding.codec are options that exist on the file source. Adding decoding options to the file source is being tracked by #9210.

If you want to parse the messages as JSON you will need to use a remap transform with the parse_json function.

Sorry, English isn't my first language, so it's hard for me to explain. Could you provide a simple example of how to use the remap transform with the parse_json function? I want to convert the message into JSON format.

for the file source, I did not find how to specify the encoding

jszwedko Dec 4, 2024
Maintainer

Correct, the file source does not support codecs like other sources do. This gap is being tracked by #9210

For the remap transform you can see the docs: https://vector.dev/docs/reference/configuration/transforms/remap/

In particular, you will want to use this function: https://vector.dev/docs/reference/vrl/examples/#parse_json

pront Dec 4, 2024
Maintainer

Here is a sample config:

sources:
  source0:
    type: file
    include:
      - "/path/to/log.json"

transforms:
  transform0:
    type: remap
    inputs:
      - source0
    source: |
      .message = parse_json!(string!(.message))

sinks:
  console:
    type: console
    inputs: ["transform0"]
    encoding:
      codec: json
      json:
        pretty: true

The magic happens at .message = parse_json!(string!(.message)) it basically takes the bytes from file (line by line) and constructs an object map. And finally it overwrites the .message with the result.

You try this on the VRL playground. See example.

Answer selected by pront

soulg24 Dec 5, 2024
Author

Вот пример конфигурации:
sources:
  source0:
    type: file
    include:
      - "/path/to/log.json"

transforms:
  transform0:
    type: remap
    inputs:
      - source0
    source: |
      .message = parse_json!(string!(.message))

sinks:
  console:
    type: console
    inputs: ["transform0"]
    encoding:
      codec: json
      json:
        pretty: true
Магия происходит в .message = parse_json!(string!(.message)): по сути, он берёт байты из файла (строка за строкой) и создаёт карту объектов. И, наконец, он перезаписывает .message результатом.

Попробуйте это на игровой площадке VRL. Смотрите Пример.

That's fantastic! Thanks for providing the examples.

soulg24 · 2025-01-14T17:11:35Z

soulg24
Jan 14, 2025
Author

I'm trying to use your example.
[sources.my_source]
type = "file"
include = ["/opt/log/create.log"]
read_from = "end"

[transforms.tr_log]
type = "remap"
inputs = [ "my_source" ]
source = """
.message = parse_json!(string!(.message))
"""
[sinks.stdout]
type = "console"
target = "stdout"
inputs = ["tr_log"]
encoding.codec = "json"

I'll get an error.

2025-01-14T17:05:05.845922Z ERROR transform{component_kind="transform" component_id=tr_log component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for "parse_json" at (11:41): unable to parse json: key must be a string at line 1 column 2" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

4 replies

jszwedko Jan 14, 2025
Maintainer

Are you sure the input is JSON? That error seems to indicate that it isn't. Can you share an example input file?

soulg24 Jan 15, 2025
Author

{"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
{"timestamp": "2024-12-04 10:11:43.037671", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
{"timestamp": "2024-12-04 10:11:43.608739", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "tech@user.com", "request param": "name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}

jszwedko Jan 15, 2025
Maintainer

Unfortunately I'm not able to reproduce this issue with this input. If I use this input with the above configuration Vector outputs:

2025-01-15T15:31:41.855832Z  INFO vector::app: Log level is enabled. level="info"
2025-01-15T15:31:41.856694Z  INFO vector::app: Loading configs. paths=["/tmp/vector.toml"]
2025-01-15T15:31:41.902647Z  INFO vector::topology::running: Running healthchecks.
2025-01-15T15:31:41.902791Z  INFO vector: Vector has started. debug="false" version="0.43.0" arch="aarch64" revision=""
2025-01-15T15:31:41.902811Z  INFO vector::topology::builder: Healthcheck passed.
2025-01-15T15:31:41.902815Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2025-01-15T15:31:41.902908Z  INFO source{component_kind="source" component_id=my_source component_type=file}: vector::sources::file: Starting file server. include=["/tmp/tmp.log"] exclude=[]
2025-01-15T15:31:41.904168Z  INFO source{component_kind="source" component_id=my_source component_type=file}:file_server: file_source::checkpointer: Loaded checkpoint data.
2025-01-15T15:31:41.905311Z  INFO source{component_kind="source" component_id=my_source component_type=file}:file_server: vector::internal_events::file::source: Found new file to watch. file=/tmp/tmp.log
{"file":"/tmp/tmp.log","host":"COMP-J4C4P27K9Q","message":{"Method":"POST","URL":"http://testapp:8080/create/s","level":"debug","request host":"xx.xx.xxx.xxx","request param":"name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ","request user":"tech@user.com","response":"204","timestamp":"2024-12-04 10:10:08.069925"},"source_type":"file","timestamp":"2025-01-15T15:31:41.905540Z"}
{"file":"/tmp/tmp.log","host":"COMP-J4C4P27K9Q","message":{"Method":"POST","URL":"http://testapp:8080/create/s","level":"debug","request host":"xx.xx.xxx.xxx","request param":"name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ","request user":"tech@user.com","response":"204","timestamp":"2024-12-04 10:11:43.037671"},"source_type":"file","timestamp":"2025-01-15T15:31:41.905583Z"}
{"file":"/tmp/tmp.log","host":"COMP-J4C4P27K9Q","message":{"Method":"POST","URL":"http://testapp:8080/create/s","level":"debug","request host":"xx.xx.xxx.xxx","request param":"name='t24120412582-pg14stb-001.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ","request user":"tech@user.com","response":"204","timestamp":"2024-12-04 10:11:43.608739"},"source_type":"file","timestamp":"2025-01-15T15:31:41.905586Z"}

As expected. Are you able to reproduce with the input you provided?

dpa456 Jan 16, 2025

I think it is necessary to check that there are no unnecessary values when it is entered in the file.
For example, newline

I checked the error in two cases.

If there is a newline when it is first entered

{"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "[tech@user.com](mailto:tech@user.com)", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
(newline)

If there is a new line and a new input is entered in that part

{"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "[tech@user.com](mailto:tech@user.com)", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
(newline)

-> 

{"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "[tech@user.com](mailto:tech@user.com)", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}
{"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "[tech@user.com](mailto:tech@user.com)", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}

For number 1, get an first message and empty string.

{"file":"/tmp/tmp.log","host":"COMP-J4C4P27K9Q","message":{"Method":"POST","URL":"http://testapp:8080/create/s","level":"debug","request host":"xx.xx.xxx.xxx","request param":"name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ","request user":"tech@user.com","response":"204","timestamp":"2024-12-04 10:10:08.069925"},"source_type":"file","timestamp":"2025-01-15T15:31:41.905540Z"}

{"file":"/tmp/tmp.log","host":"COMP-J4C4P27K9Q","message":""},"source_type":"file","timestamp":"2025-01-15T15:31:41.905540Z"}

For number 2, remove the first '{' and get the .message.

"timestamp": "2024-12-04 10:10:08.069925", "level": "debug", "Method": "POST", "URL": "http://testapp:8080/create/s", "request host": "xx.xx.xxx.xxx", "request user": "[tech@user.com](mailto:tech@user.com)", "request param": "name='vmtest241204.jmeter-test.user.com' os='Ubuntu 20.04 LTS' ipaddress='xx.xx.xx.xx' ", "response": "204"}

soulg24 · 2025-01-17T08:00:30Z

soulg24
Jan 17, 2025
Author

An error occurs in the following code:

source = """
.message = parse_json!(string!(.message))
"""
When I use the configuration from the example you provided in YAML format, it works correctly, as shown below:

source: |
.message = parse_json!(string!(.message))
However, when I switch to my TOML configuration, errors arise, even though the VRL (Vector Remap Language) format remains unchanged.

Let me know if you need further clarification!

2 replies

pront Jan 17, 2025
Maintainer

Please provide both configs in full.

soulg24 Jan 22, 2025
Author

YAML

sources:
source0:
type: file
include:
- "/opt/log/log.json"

transforms:
transform0:
type: remap
inputs:
- source0
source: |
.message = parse_json!(string!(.message))

sinks:
console:
type: console
inputs: ["transform0"]
encoding:
codec: json
json:
pretty: true

TOML

[sources.my_source]
type = "file"
include = ["/opt/log/log.json"]
read_from = "end"

[transforms.tr_log]
type = "remap"
inputs = [ "my_source" ]
source = """
.message = parse_json!(string!(.message))
"""

[sinks.stdout]
type = "console"
target = "stdout"
inputs = ["tr_log"]
encoding.codec = "json"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log data in JSON format #21915

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 11 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

log data in JSON format #21915

soulg24 Nov 29, 2024

Replies: 4 comments · 11 replies

pront Dec 2, 2024 Maintainer

soulg24 Dec 4, 2024 Author

jszwedko Dec 4, 2024 Maintainer

soulg24 Dec 4, 2024 Author

jszwedko Dec 4, 2024 Maintainer

pront Dec 4, 2024 Maintainer

soulg24 Dec 5, 2024 Author

soulg24 Jan 14, 2025 Author

jszwedko Jan 14, 2025 Maintainer

soulg24 Jan 15, 2025 Author

jszwedko Jan 15, 2025 Maintainer

dpa456 Jan 16, 2025

soulg24 Jan 17, 2025 Author

pront Jan 17, 2025 Maintainer

soulg24 Jan 22, 2025 Author

soulg24
Nov 29, 2024

Replies: 4 comments 11 replies

pront
Dec 2, 2024
Maintainer

soulg24 Dec 4, 2024
Author

jszwedko
Dec 4, 2024
Maintainer

soulg24 Dec 4, 2024
Author

jszwedko Dec 4, 2024
Maintainer

pront Dec 4, 2024
Maintainer

soulg24 Dec 5, 2024
Author

soulg24
Jan 14, 2025
Author

jszwedko Jan 14, 2025
Maintainer

soulg24 Jan 15, 2025
Author

jszwedko Jan 15, 2025
Maintainer

soulg24
Jan 17, 2025
Author

pront Jan 17, 2025
Maintainer

soulg24 Jan 22, 2025
Author