Filebeat : Split the multiline json(array of json object) from message field to separate fields

[mageshsankar]

I need to use filebeat to push my json data into elastic search, but I'm having trouble decoding my json fields into separate fields extracted from the message field.

Filebeat version : 7.16.2

Filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /logs/*.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: "2"
      target: ""
      overwrite_keys: true
      add_error_key: false

output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true
  hosts: ["http://localhost:9200"] 

Json Input :

{
  "Source": [
    {
      "date": "28-09-2021",
      "language": " C++",
      "comment": 11,
      "code": 150325
    },
    {
      "date": "28-09-2021",
      "language": " C++",
      "comment": 11,
      "code": 106026
    }
  ]
}

Current Output:

Expected Output:
separate fields

_source: {
@timestamp: "2022-01-12T09:12:36.904Z",
"date": "28-09-2021",
 "language": " C++",
 "comment": 11,
  "code": 106026
input: {
type: "log"
},

Please suggest to decode the multiline json in filebeat?

[jsoriano]

I guess these comments also apply here: #13137

It can be tricky to store arrays of objects in single events.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic]

This issue doesn't have a Team:<team> label.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!