Metricbeats http module does gives error for parsing json.

[syedsfayaz]

I am using metricbeat to parse jenkins build parameters. I am using http module. But I am seeing error.

Please post all questions and issues on https://discuss.elastic.co/c/beats
https://discuss.elastic.co/t/error-metricbeat-sending-json-data-to-elastic-search/192582

Also tried the solution in the github link which does not work.
#8292

Please include configurations and logs if available.

For confirmed bugs, please report:

• Version: 7.2
• Operating System: RehHat
• Discuss Forum URL: https://discuss.elastic.co/t/error-metricbeat-sending-json-data-to-elastic-search/192582
• Steps to Reproduce:

[jsoriano]

Hi @syedsfayaz,

I see in the topic in discuss that you managed to get the event, but it had some unparsed fields. This actions field is a list of objects, what can be problematic to store in Elasticsearch, see the note about arrays of objects in the docs.

Were the rest of fields parsed correctly?

[syedsfayaz]

Not all the fields parsed correctly. As you see the actions fields are not parsed. As mentioned in this link. https://www.elastic.co/guide/en/elasticsearch/reference/7.3/nested.html Can you guide me how I can use this nested data type. Do I need to add these settings in http.yml?

I need to parse these values from the json apart from other results.
"parameters" : [
{
"_class" : "hudson.model.StringParameterValue",
"name" : "BUILDNUMBER",
"value" : "12571"

[jsoriano]

Yeah, I think that all fields that are arrays of objects, as actions, or this parameters, are not going to be correctly handled by Beats and/or Elasticsearch.

The use of nested types may need further development, because they are not completely supported by Beats and Kibana.

I think there is no good solution at the moment for the data you want to collect.

One option to support it would be to handle arrays of objects in the http module, but it is not clear what should be done in this case, and how to do it in a generic way. Another option would be to implement a jenkins module, or community beat, to collect this information.

[syedsfayaz]

Can Logstash parse this data?

[jsoriano]

Can Logstash parse this data?

I think that with logstash the problematic is similar, it is not clear what to do with fields that are lists of objects.

[syedsfayaz]

I mean I can write some curl command or some python/power shell script to get what exactly values I need from the jenkins api and then pass those to log-stash.

[jsoriano]

@syedsfayaz if you use an external script to create the event with the values you want, you can also use the HTTP server metricset of Metricbeat to send these events to Elasticsearch.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!