Fixes parsing of @timestamp for Elasticsearch Audit JSON logs

[lucabelluccini]

Fixes parsing of @timestamp for Elasticsearch Audit JSON logs.

E.g. of audit JSON logs:

{"@timestamp":"2019-09-05T14:02:37,921", "node.id":"UwRu4mReRtyJO1-FWAPvIQ", "event.type":"transport", "event.action":"authentication_success", "user.name":"_system", "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "realm":"__fallback", "request.id":"474ZciqtQteOhjLO3OdZIw", "action":"indices:monitor/stats", "request.name":"IndicesStatsRequest"}

Related to #13367

Would it be possible to review @jsoriano @ycombinator ?

[jsoriano]

@lucabelluccini thanks for opening this PR, it seems that it is failing for our current test files. Most of them seem to be ISO8601 dates with timezone included, but we have some of them without timezone. I wonder if the ones without timezones are from older versions.

From what version is the log entry you posted?

[elasticmachine]

Pinging @elastic/stack-monitoring

[lucabelluccini]

Hello @jsoriano, this is Filebeat 7.1.1 with Elasticsearch 7.1.1.
The timezone conversion was not working there.
I'll take a look with Filebeat 7.1.1 with ES 7.3.1.
Maybe we can specify multiple formats so we can cover different cases?

[ycombinator]

Hi @lucabelluccini, I'm able to reproduce the issue you are seeing with Elasticsearch 7.1.1. This is because in that version of ES the audit logs (JSON format) contain a @timestamp field which does not contain a time zone. However, in later versions of ES the audit logs (JSON format) instead contain a timestamp (notice the lack of @ prefix) field which does contain a timestamp. 🤦‍♂🤦‍♀

Thanks for bringing this to our attention with this PR. However, the fixes in this PR are not compatible across all versions of ES, as @jsoriano pointed out in his comment above.

After much experimentation (and learning a few special cases in Painless syntax 😅) I believe I finally have the correct configuration necessary for the elasticsearch/audit fileset to parse audit logs (JSON format) from all supported ES versions correctly. I've put up this change in #15942. Please take a look at that PR and if you're happy with it, I think we can close this one out in favor of #15942. Thanks!

[ycombinator]

#15942 has already been reviewed and merged now. So I'm closing this PR unmerged in favor of #15942.