Add a message field to messages that have a JSON error_key

[Cylindric]

When using FileBeat to ingest records formatted as JSON, it is useful to have the ones that failed to parse logged, which is achieved with the add_error_key option.

Would it be possible to add the actual line that failed to the document as well? Currently all we have to go on is to search 10Gb/day of log files for a miss-placed comma. There are many commas in a JSON file :)

This would be similar to how Logstash can add the unprocessed "message" field to each document.

Error decoding JSON: invalid character ',' in numeric literal

It does have the offset, but as these files are frequently rotated, it is not easy to find that specific offset in the particular file that has been rotated out.

[ruflin]

Can you share the config you are using and a line that causes the above issue? This will make it easier to test this potential enhancement. Probably it requires some changes to the json reader.

[Cylindric]

The error message is accurate, in that my log files may well contain some malformed JSON. My question is one of finding out what, so I can fix the reporting app.

[ruflin]

@Cylindric Sorry, I think my question above was not clear. Fully agree that there is no issue with the config, it is only easier to have on in the issue already when someone tries to reproduce it + some potential log line you already have which is broken.

[Cylindric]

Any line that isn't valid JSON would correctly cause a parse error, I guess.

{ "message": I am a 'banana" }

[ismael-hasan]

Hi @ruflin ,

When enabling json.add_error_key the original failing message is not shown; when that property is not set, the original message is shown. I think that the behavior should be consistent, regardless of that option (and, preferably, including the failing line). It follows a test case:

Input (incorrect) document:

{ "field1":"value1", field2:"value2" }

Configuration and output when json.add_error_key is set to false (output includes the failing line):

filebeat.prospectors:

- input_type: log
  paths: 
  - 'C:\\tmp\\test.log'

  json.keys_under_root: true
  json.add_error_key: false

#================================ Fields ====================================
fields_under_root: true

output.console:
  enabled: true
  pretty: true

{
  "@timestamp": "2018-05-04T06:43:19.714Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "C:\\tmp\\test.log",
  "offset": 40,
  "json": {},
  "message": "{ \"field1\":\"value1\", field2:\"value2\" }",
  "beat": {
    "name": "DESKTOP-I0VERPV",
    "hostname": "DESKTOP-I0VERPV",
    "version": "6.2.4"
  }
}

Configuration and output when json.add_error_key is set to true (output does not include the failing line):

filebeat.prospectors:

- input_type: log
  paths: 
  - 'C:\\tmp\\test.log'

  json.keys_under_root: true
  json.add_error_key: true

#================================ Fields ====================================
fields_under_root: true

output.console:
  enabled: true
  pretty: true

{
  "@timestamp": "2018-05-04T06:43:55.688Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "C:\\tmp\\test.log",
  "offset": 40,
  "error": {
    "message": "Error decoding JSON: invalid character 'f' looking for beginning of object key string",
    "type": "json"
  },
  "beat": {
    "hostname": "DESKTOP-I0VERPV",
    "version": "6.2.4",
    "name": "DESKTOP-I0VERPV"
  }
}

[andrewkroh]

Because the message content is lost on error I think we should prioritize this issue.

Another odd thing is the "json": {}, that shows up in the event.

[andrewkroh]

I think this has been addressed by #6591 and is a duplicate of #6516.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.