[Security Solutions] (Phase 1) Adds an application cache called metrics entities and integrates it within Security Solutions behind a feature flag #96446

FrankHassanabad · 2021-04-07T15:37:32Z

Summary

Phase 1 of a multi-phase cautious approach for adding an experimental application cache for Kibana solutions called metric_entities and integrates it within Security Solutions.

Phase 1 is putting experimental support into the application without breaking existing features. Lots of TODO's, conversations and a possible RFC from phase 1 to phase 2 approach. Some features are missing, but for phase 1 the general idea and code is all there.

To enable this first phase after checking out the branch add this to your kibana.dev.yml

xpack.metricsEntities.enabled: true
xpack.securitySolution.enableExperimental: ['metricsEntitiesEnabled']

Then go into Stack Management -> Advanced Settings (Under Security Solutions) and set the enabled to true like so:

Next go to the security_solutions page and you will see it being activated and you will have these transforms running if you look under stack management:

On the hosts page, network, page, etc... You can see them being activated when you have no query/filter and you click on request:

You will see in the request the index patterns all starting with estc_xyz*

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

We have lots of TODO's but no concrete docs with this just yet.

Documentation was added for features that require explanation or tutorials

Behind a feature flag and this isn't there yet.

Unit or functional tests were updated or added to match the most common scenarios

… the network screen

…p and TODO's)

…re natural

… blocks

…ng it by default for now

… experimental feature of metricsEntity

elasticmachine · 2021-04-29T19:47:36Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

jbudz

get_server_watch_paths LGTM

spong · 2021-04-30T16:40:48Z

x-pack/plugins/metrics_entities/README.md

+Restart Kibana and ensure that you are using `--no-base-path` as changing the base path is a feature but will
+get in the way of the CURL scripts written as is.


Platform discourages --no-base-path. We should adapt our scripts to play nice when without this flag. We've discussed this previously with some options being leveraging a static server.basePath: '/kbn', or wrapping our scripts in a helper that calls curl -L http://localhost:5601/ and looks at the redirect and then uses that.

Thanks for the tip

kibanamachine · 2021-04-30T17:08:59Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2013	2029	+16

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`metricsEntities`	-	5	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	6.5MB	6.6MB	+32.3KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`metricsEntities`	-	1	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	160.0KB	160.8KB	+849.0B

Unknown metric groups

API count

id	before	after	diff
`metricsEntities`	-	8	+8

History

💔 Build #123397 failed 2270709
💚 Build #123334 succeeded 06d87df
💛 Build #123273 was flaky bd49ce3
💚 Build #123118 succeeded 4ad83a3
💚 Build #123094 succeeded 71b4f96

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @FrankHassanabad

spong

Checked out, tested locally, and pair code-reviewed with @FrankHassanabad. Thank you for all your efforts here, and for taking the time to go over all these changes with me -- this is giant leap in providing fast and snappy UI's via metrics and entities! LGTM! 👍 🙂 🚀

…cs entities and integrates it within Security Solutions behind a feature flag (elastic#96446) ## Summary Phase 1 of a multi-phase cautious approach for adding an experimental application cache for Kibana solutions called `metric_entities` and integrates it within Security Solutions. Phase 1 is putting experimental support into the application without breaking existing features. Lots of TODO's, conversations and a possible RFC from phase 1 to phase 2 approach. Some features are missing, but for phase 1 the general idea and code is all there. To enable this first phase after checking out the branch add this to your `kibana.dev.yml` ```yml xpack.metricsEntities.enabled: true xpack.securitySolution.enableExperimental: ['metricsEntitiesEnabled'] ``` Then go into Stack Management -> Advanced Settings (Under Security Solutions) and set the enabled to true like so: <img width="1229" alt="Screen Shot 2021-04-08 at 2 21 02 PM" src="https://user-images.githubusercontent.com/1151048/114091276-b3cbb700-9875-11eb-9083-5c1d91dd20ed.png"> Next go to the security_solutions page and you will see it being activated and you will have these transforms running if you look under stack management: <img width="1710" alt="Screen Shot 2021-04-29 at 2 00 27 PM" src="https://user-images.githubusercontent.com/1151048/116611174-4a2e4e00-a8f3-11eb-9e15-55cb504dfb2a.png"> On the hosts page, network, page, etc... You can see them being activated when you have no query/filter and you click on request: <img width="1405" alt="Screen Shot 2021-04-29 at 2 01 28 PM" src="https://user-images.githubusercontent.com/1151048/116611274-6a5e0d00-a8f3-11eb-9998-9f5b3d1c5c63.png"> You will see in the request the index patterns all starting with `estc_xyz*` ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) We have lots of TODO's but no concrete docs with this just yet. - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials Behind a feature flag and this isn't there yet. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

kibanamachine · 2021-04-30T18:38:49Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…cs entities and integrates it within Security Solutions behind a feature flag (#96446) (#98967) ## Summary Phase 1 of a multi-phase cautious approach for adding an experimental application cache for Kibana solutions called `metric_entities` and integrates it within Security Solutions. Phase 1 is putting experimental support into the application without breaking existing features. Lots of TODO's, conversations and a possible RFC from phase 1 to phase 2 approach. Some features are missing, but for phase 1 the general idea and code is all there. To enable this first phase after checking out the branch add this to your `kibana.dev.yml` ```yml xpack.metricsEntities.enabled: true xpack.securitySolution.enableExperimental: ['metricsEntitiesEnabled'] ``` Then go into Stack Management -> Advanced Settings (Under Security Solutions) and set the enabled to true like so: <img width="1229" alt="Screen Shot 2021-04-08 at 2 21 02 PM" src="https://user-images.githubusercontent.com/1151048/114091276-b3cbb700-9875-11eb-9083-5c1d91dd20ed.png"> Next go to the security_solutions page and you will see it being activated and you will have these transforms running if you look under stack management: <img width="1710" alt="Screen Shot 2021-04-29 at 2 00 27 PM" src="https://user-images.githubusercontent.com/1151048/116611174-4a2e4e00-a8f3-11eb-9e15-55cb504dfb2a.png"> On the hosts page, network, page, etc... You can see them being activated when you have no query/filter and you click on request: <img width="1405" alt="Screen Shot 2021-04-29 at 2 01 28 PM" src="https://user-images.githubusercontent.com/1151048/116611274-6a5e0d00-a8f3-11eb-9998-9f5b3d1c5c63.png"> You will see in the request the index patterns all starting with `estc_xyz*` ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) We have lots of TODO's but no concrete docs with this just yet. - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials Behind a feature flag and this isn't there yet. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>

## Summary Removes the `metrics_entities` plugin and POC. As a different direction will be taken and people can look back at the git history for it as they see fit if they need to refer to it. Once it's re-added it it will be through an RFC process and re-discussed. Earlier PR's which added the POC: #96446 #104559 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

FrankHassanabad added 16 commits March 29, 2021 17:26

Initial framework code in place with lots of TODO's, all WIP.

dcd3276

Merge branch 'master' into add-summary-metrics-plugin

9f84e84

Added a new module for all of security solutions and more metrics for…

f93e293

… the network screen

Added delete security_solutions and more notes to the README.md

5922ead

Initial integration with the web application for querying (lots of wi…

66c6e21

…p and TODO's)

Merge branch 'master' into add-summary-metrics-plugin

33ef3cf

More bug fixes, changed out the names for smaller names of things

02d2acd

Renamed plugin to be metrics_entities

f55e528

Renamed more files to use entities instead of summaries as that is mo…

9f28076

…re natural

More renaming to keep things more concise

fed2036

Renamed

ed2ceb6

Merge branch 'master' into add-summary-metrics-plugin

ef1ccb8

Remove merged things that shouldn't be in tsconfig

04d22ba

Remove extra line

179d6d1

Cleanup and more todo blocks around areas

d4652ff

Pushed the timerange back upwards to the front end, removed more TODO…

5e90728

… blocks

FrankHassanabad self-assigned this Apr 7, 2021

FrankHassanabad added 2 commits April 7, 2021 17:12

Updated the i18n key to work out

0a28e26

Added the matrix histogram that was missing from the refactor

0a99e53

FrankHassanabad changed the title ~~Add summary metrics plugin~~ Adds an application cache for Kibana solutions called metrics entities and integrates it with Security Solutions. Apr 7, 2021

FrankHassanabad added 6 commits April 7, 2021 17:23

Removed accidental change from cases

45465a8

Added feature flags to enable/disable the metrics_summary and disabli…

6d29a66

…ng it by default for now

Added experimental flag support to the ui_settings now as well

458e7e8

Fixes unit tests to work with experimental and thus ... tada ... this…

408d0eb

… experimental feature of metricsEntity

Fixed server watch path jest test

ec09f42

Merge branch 'master' into add-summary-metrics-plugin

19eed74

FrankHassanabad changed the title ~~Adds an application cache for Kibana solutions called metrics entities and integrates it with Security Solutions.~~ Adds an application cache for Kibana solutions called metrics entities and integrates it with Security Solutions Apr 12, 2021

FrankHassanabad added 2 commits April 12, 2021 22:01

Fixed error with jests from update from master

1d07694

Merge branch 'master' into add-summary-metrics-plugin

2cf992e

madirey mentioned this pull request Apr 13, 2021

[Security Solution] Network/DNS tab results in too_many_buckets_exception #97004

Closed

FrankHassanabad marked this pull request as ready for review April 29, 2021 19:46

FrankHassanabad requested review from a team as code owners April 29, 2021 19:46

FrankHassanabad changed the title ~~Adds an application cache for Kibana solutions called metrics entities and integrates it with Security Solutions~~ [Security Solutions] Adds an application cache called metrics entities and integrates it within Security Solutions Apr 29, 2021

FrankHassanabad added the Team:Detections and Resp Security Detection Response Team label Apr 29, 2021

FrankHassanabad added release_note:feature Makes this part of the condensed release notes v8.0.0 v7.14.0 auto-backport Deprecated - use backport:version if exact versions are needed labels Apr 29, 2021

jbudz approved these changes Apr 29, 2021

View reviewed changes

FrankHassanabad added 3 commits April 29, 2021 15:54

Updated README and some notes

2270709

Added docs that wanted to be updated

d5ab054

Merge branch 'master' into add-summary-metrics-plugin

8c3e322

spong reviewed Apr 30, 2021

View reviewed changes

spong approved these changes Apr 30, 2021

View reviewed changes

FrankHassanabad merged commit af228f0 into elastic:master Apr 30, 2021

FrankHassanabad deleted the add-summary-metrics-plugin branch April 30, 2021 18:36

kibanamachine mentioned this pull request Apr 30, 2021

[7.x] [Security Solutions] (Phase 1) Adds an application cache called metrics entities and integrates it within Security Solutions behind a feature flag (#96446) #98967

Merged

FrankHassanabad added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:feature Makes this part of the condensed release notes labels Jul 22, 2021

FrankHassanabad mentioned this pull request Apr 7, 2022

[Security Solutions] Removes POC transforms #129673

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solutions] (Phase 1) Adds an application cache called metrics entities and integrates it within Security Solutions behind a feature flag #96446

[Security Solutions] (Phase 1) Adds an application cache called metrics entities and integrates it within Security Solutions behind a feature flag #96446

FrankHassanabad commented Apr 7, 2021 •

edited

Loading

elasticmachine commented Apr 29, 2021

jbudz left a comment

spong Apr 30, 2021

FrankHassanabad Apr 30, 2021

kibanamachine commented Apr 30, 2021

API count

spong left a comment

kibanamachine commented Apr 30, 2021

		Restart Kibana and ensure that you are using `--no-base-path` as changing the base path is a feature but will
		get in the way of the CURL scripts written as is.

[Security Solutions] (Phase 1) Adds an application cache called metrics entities and integrates it within Security Solutions behind a feature flag #96446

[Security Solutions] (Phase 1) Adds an application cache called metrics entities and integrates it within Security Solutions behind a feature flag #96446

Conversation

FrankHassanabad commented Apr 7, 2021 • edited Loading

Summary

Checklist

elasticmachine commented Apr 29, 2021

jbudz left a comment

Choose a reason for hiding this comment

spong Apr 30, 2021

Choose a reason for hiding this comment

FrankHassanabad Apr 30, 2021

Choose a reason for hiding this comment

kibanamachine commented Apr 30, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

History

spong left a comment

Choose a reason for hiding this comment

kibanamachine commented Apr 30, 2021

💚 Backport successful

FrankHassanabad commented Apr 7, 2021 •

edited

Loading