[Fleet] Update Elastic Agent dashboard, package icon, and data stream mappings

[nchaulet]

Description

Resolve #951
Resolve elastic/beats#27405
Resolve #1453

Update the elastic agent dashboard with more metrics:

• CGroup CPU usage
• filebeat and metricbeat events rate, throughput and errors

Data streams changes:

• Add the missing beat* fields to the metrics data_stream
• Add back the log.level field to fix a bug with the log level selection in the agent logs page in Fleet

UI Change

cc @ravikesarwani

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-08-18T18:06:32.134+0000

• Duration: 12 min 48 sec

• Commit: 89c3485

Test stats 🧪

Test	Results
Failed	0
Passed	30
Skipped	0
Total	30

Trends 🧪

[ravikesarwani]

@nchaulet I don't see cgroup memory information. Can you please add that as well. Both CPU and memory limits are critical in the container (Docker)environment and showing cgroup information would be very useful there.

[ravikesarwani]

@matschaffer I know you have some great field experience and love to hear any feedback from your side on this Elastic Agent dashboard.

[nchaulet]

Updated with the memory Cgroup limit

[ruflin]

We add a lot of fields to each template. Do we really need all of them? In general we should only map the ones we need in the queries for the dashboards. For all the other ones runtime fields could be used if needed.

@nimarezainia Would be great if you could spin this up and have a look if all works as expected.
@nchaulet Thx for taking it on.

[nchaulet]

We add a lot of fields to each template. Do we really need all of them? In general we should only map the ones we need in the queries for the dashboards. For all the other ones runtime fields could be used if needed.

We probably do not need any of them, I am going to use runtime fields instead

[nchaulet]

@ruflin Actually looks like we do not support runtime fields in a package for now (elastic/package-spec#39) just going to cleanup the fields to only add the one needed in the dashboard

[ravikesarwani]

@nchaulet We haven't added all the fields in the dashboard that the current beats shows. the idea was to make enough things available but users can easily extend the dashboard where needed.Can we make sure some of the missing fields in Events rate, Throughput (Bytes received), System load are available to user for easily extending the dashboard?

[ruflin]

I did not mean runtime fields on the packages / templates but on query time.

[nchaulet]

I did not mean runtime fields on the packages / templates but on query time.

Maybe I am saying a mistake here, but I think the only way to use runtime fields in a dashboard or a visualization is to add the runtime field to the index pattern

[nchaulet]

@nchaulet We haven't added all the fields in the dashboard that the current beats shows. the idea was to make enough things available but users can easily extend the dashboard where needed.Can we make sure some of the missing fields in Events rate, Throughput (Bytes received), System load are available to user for easily extending the dashboard?

I added the mappings for all the events, and throughput even the one we do not use in the dashboard.

[jen-huang]

@hop-dev was looking at elastic/beats#27404 and found that event.dataset is missing mappings too. @nchaulet do your mapping changes here add that back? It looks like most other packages have this field defined in base-fields.yml files:

integrations/packages/nats/data_stream/connection/fields/base-fields.yml

Line 17 in 0732c78

- name: event.dataset

[nchaulet]

@hop-dev was looking at #27404 and found that event.dataset is missing mappings too. @nchaulet do your mapping changes here add that back? It looks like most other packages have this field defined in base-fields.yml files:

It does not but I can try to add it back and see if it solve the Fleet issue.

[matschaffer]

@nchaulet is there a doc that'll teach me how to get this running live so I can see what's available?

A couple of thoughts just based on the screenshot:

• cgroup info will be important for people running containers with limits, but might just take up screen space for people who don't. Not sure if we have options for toggling visibility, but it might be good to consider for future work.

• it'd be good to get info about restarts (for example cgroup OOMs) or start up errors, not just output errors. Typically I'd read cgroup ooms from the OS system log, so not sure how easy that will be. But without it we could have a case where the cgroup memory limit is too small and we just can't start. Would be good to make the failure obvious to the user.

• The titles are good, but it'd be even better to have descriptions on each viz as well so you can know exactly what metric is being shown without having to edit the viz (especially rough for TSVB since filters will drop away when you edit the viz).

[ruflin]

@matschaffer Here some docs on how to get this setup for testing: https://github.com/elastic/integrations/blob/master/docs/developer_workflow_design_build_test_integration.md

[ruflin]

@nchaulet You are right about the Kibana index pattern and runtime fields. I was thinking the user would update these to use it. But these are global and not per dashboard and I think there is also a chance to we will overwrite it? I'm hoping in the future each Dashboard will have its own "data view / kibana index pattern" stored locally so it does not impact each other and we do not overwrite it. I know there is some work ongoing in this direction on the Kibana side. Maybe @joshdover has some more specifics.

[nchaulet]

Just updated with the description

[jen-huang]

@nchaulet Do you think the dashboard changes here also fixes #1453?

[jen-huang]

@nchaulet Nice thanks for adding event.dataset. #1453 is a different issue about CPU usage calculation being incorrect.

[nchaulet]

@jen-huang My bad I misread your comment, I just pushed a fix for #1453

[nimarezainia]

Hi, I can't comment on the individual fields but based on the dashboard this looks good. thanks.

[kpollich]

LGTM in Fleet/Integrations UI's. 🚀

[dikshachauhan-qasource]

Hi @nchaulet

We have attempted to validate above ticket on 7.15 BC2 Kibana QA build and could not find required changes on elastic agent metrics dashboard for default docker-fleet-agent. Though we have enabled logs collection settings for default fleet server agent.

Screenshot:

@nchaulet Could you Please let us know how to validate above changes on dashboard.

Working fine:

• On Integration tab, elastic agent package icon and sub-headings were correctly updated as per mock screenshot.

cc @EricDavisX

Thanks
QAS

[amolnater-qasource]

Hi @EricDavisX
We have revalidated this issue on 7.15.0 BC-2 Kibana Cloud environment and found it fixed now.

• Log level dropdown is showing user options to select log level say- error, info, debug, warn etc.

Build details:

Build: 43742
Commit: 113d5f2a745ab9be7d567d13ff906dfd0f5b7a9a

Screenshot:

Thanks
QAS

[dikshachauhan-qasource]

Hi @nchaulet

We have validated above PR again on 7.15BC4 cloud-qa build and came up with following observations shared above on Elastic-agent package version 1.2.0. Please let us know if we are missing anything in validating above.

Observations:

1. Icons is now visible on Integration detail page and data stream page.
   

2. Data streams are now available for filebeat and metricbeat.
   

3. On Dashboard no data is available for Host selected by default in filter value.
   

4. However, on removing pre-selected host value on dashboard data shows up on graphs for elastic-agent.
   

Query: No data is available on graphs Cgroup CPUusage and [Elastic Agent] Errors in writing the response from the output. Is it expected.

Thanks
QAS

[nchaulet]

@dikshachauhan-qasource thanks for the testing

The default filter issue will be fixed by #1569

For the no data for cgroup and errors it's expected, you will have data in cgroup if you run the agent in a container with memory and cpu limit. and errors if there are output errors.

[dikshachauhan-qasource]

Hi @nchaulet

Thanks for the feedback.

Further, we revalidated fixes for #1569 under latest snapshot build from cloud-qa. However, as merges were not found under same, so we will validate same on next BC.

Build details:
BUILD 43910
COMMIT 3a43cf52fb7fb41403239132e78fd43ab9a0e118

Thanks
QAS