[ECS] 7.0.0-rc1 Broken Beats Dashboards

[EthanStrider]

Certain Beats dashboards appear to be broken by ECS changes since they still reference the old data field names.

There are probably more... I'll update this ticket as I find them.

For confirmed bugs, please report:

• Version: 7.0.0-rc1
• Operating System: Cloud

[EthanStrider]

Ok, seems like there are a lot of broken dashboards. I'm thinking about writing a script to search all the old terms against this repo: https://github.com/elastic/kibana/tree/7.0/x-pack/plugins/ml/server/models/data_recognizer/modules

[ruflin]

For the Beats dashboards we were running this script to migrate the fields: https://github.com/elastic/beats/blob/master/script/kibana-migration.py That should also work for the ML dashboards. But the dashboards you link above, are these from ML or the ones loaded through Beats?

[EthanStrider]

@ruflin I think it's a combination of ML dashboards and dashboards loaded through Beats. I'm working on upgrading the demo.elastic.co demo environment, so we use everything.

[EthanStrider]

From the docs here: https://www.elastic.co/guide/en/beats/libbeat/7.0/breaking-changes-7.0.html

Isn't this backwards?

Shouldn't it be:

auditd.message_type --> event.type

[fearful-symmetry]

Yah, auditd.message_type seems backwards.

So, I'm just gonna preface this by saying I have almost no experience with the kibana side of things, but I'm trying to learn. That being said, I'm looking at the nginx dashboards, and it looks like a few fields weren't hit by #9998.

From what I can tell, the script will only change names if alias: True which isn't the case for a lot of the user agent fields in ecs-migration.yml

beats/dev-tools/ecs-migration.yml

Line 1260 in d026281

alias: false

This might be why a lot of the visualizations that rely on user_agent are using the pre-ecs fields:

beats/filebeat/module/nginx/_meta/kibana/7/dashboard/Filebeat-nginx-overview.json

Line 114 in d026281

"field": "nginx.access.user_agent.os_major",

that should be user_agent.os.version for ecs?

I wonder if this is a case where we need to start changing dashboards by hand.

[EthanStrider]

that should be user_agent.os.version for ecs?

Yes, that is correct.

[EthanStrider]

Created a PR here: #11520

[ruflin]

@EthanStrider As #11520 is closed, I think we can close this issue.

@fearful-symmetry Could you do some test checks if we miss some more fields which don't have an alias? I thought I did but seems like I missed some :-(

[EthanStrider]

@ruflin Thanks! Is there going to be an updated build for 7.0.0-rc1?
https://staging.elastic.co/7.0.0-rc1-2fe9f018/summary-7.0.0-rc1.html

I'm not sure how the build system works, but I'd like to grab the docker image for Filebeat 7.0.0-rc1 with the latest fixes.

[webmat]

@EthanStrider For the event.type change, I know it looks backwards, but this is actually expected.

event.type will come back with a very well defined set of values that need to be used across all sources. So whatever was in there, populated by prior versions of Auditbeat's auditd module is being moved to a custom field now.

cc @fearful-symmetry

[ruflin]

@webmat So the change in this PR to event.type should be reverted? I'm now also confused :-)

[webmat]

event.type right now is reserved. Must not be used in 7.0.

Still going through the PR, but yes, it needs to be made auditd.event_type again

[EthanStrider]

@ruflin Sounds like just the change to the doc needs to be reverted. I added a comment.

[fearful-symmetry]

Trying to track where we are with this. We have
#11520
Related, #11538
#11545

I think those PRs should cover everything?

[webmat]

As far as I know, everything here has been addressed.

WDYT @EthanStrider?

[EthanStrider]

Looks good! If I find anything else, I'll let you know.

[ruflin]

Thanks everyone, closing this for now.