Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Observability][SecuritySolution] Update entity manager to support extension of mappings and ingest pipeline #188410

Merged
merged 7 commits into from
Jul 22, 2024
Merged

[Observability][SecuritySolution] Update entity manager to support extension of mappings and ingest pipeline #188410

merged 7 commits into from
Jul 22, 2024

Conversation

machadoum
Copy link
Member

@machadoum machadoum commented Jul 16, 2024 •
edited
Loading

Summary

Acceptance Criteria

  • When starting Kibana, the global entity index templates are no longer created
  • When installing a definition, an index template is generated and installed scoped to the definition ID
  • When deleting a definition, the related index template is also deleted
  • The index template composes the current component templates (base, entity, event) as well as the new custom component templates with the setting ignore_missing_component_templates set to true
  • The new component templates should be named: <definition_id>@platform, <definition_id>-history@platform, <definition_id>-latest@platform, <definition_id>@Custom, <definition_id>-history@custom and <definition_id>-latest@custom
  • The ingest pipelines include a pipeline processor that calls out the pipelines named <definition_id>@platform and <definition_id>-history@platform or <definition_id>-latest@platform, <definition_id>@Custom and <definition_id>-history@custom or <definition_id>-latest@custom if they exist
  • The index template should have a priority of 200 and be set to managed
  • The @Custom component template should take precedence over the @platform component template, allowing users to override things we have set if they so wish
  • set managed_by to 'elastic_entity_model',

Checklist

@machadoum machadoum requested a review from hop-dev July 16, 2024 09:44
@machadoum machadoum self-assigned this Jul 16, 2024
@machadoum machadoum added release_note:skip Skip the PR/issue when compiling release notes Team:Entity Analytics Security Entity Analytics Team labels Jul 16, 2024
@machadoum
Copy link
Member Author

/ci

1 similar comment
@machadoum
Copy link
Member Author

/ci

@machadoum machadoum changed the title Siem eem 104 [Observability][SecuritySolution] Update entity manager to support extension of mappings and ingest pipeline Jul 16, 2024
@machadoum machadoum marked this pull request as ready for review July 16, 2024 15:10
@machadoum machadoum requested a review from a team as a code owner July 16, 2024 15:10
@machadoum
Copy link
Member Author

/ci

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

@botelastic botelastic bot added the ci:project-deploy-observability Create an Observability project label Jul 18, 2024
@klacabane
Copy link
Contributor

We're now managing template at the entity definition level so we need to update the api key permissions for built in definitions. Can we please add manage_index_templates here ?

@klacabane
Copy link
Contributor

klacabane commented Jul 18, 2024 •
edited
Loading

Wondering about customization of the builtin definitions, is this something that we want or should we recommend to duplicate the definition to a custom one instead ? cc @simianhacker @tommyers-elastic

@machadoum
Copy link
Member Author

@klacabane Thank you for the great feedback! 🙇

I have updated the code according to the suggestions, except for this comment.
I don't understand the question, is it for me?

Could you please take a second look?

@elasticmachine
Copy link
Contributor

elasticmachine commented Jul 19, 2024 •
edited
Loading

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @machadoum

@klacabane
Copy link
Contributor

LGTM, thanks for the change!

I don't understand the question, is it for me?

no you don't have to worry about this one :)

@klacabane
Copy link
Contributor

There's still two open questions that we can take as follow ups

  • should we allow customization of builtin definition at the risk of breaking the solutions that are built upon them
  • what value should we provide as the managed_by meta assigned to a definition's components ?

hop-dev
hop-dev approved these changes Jul 22, 2024
View reviewed changes
Copy link
Contributor

@hop-dev hop-dev left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work 🚀 lets get answers to the 2 outstanding questions from Observability before merging.

Edit: Ignore me, I miseed " that we can take as follow ups" 👍 merge away

@hop-dev
Copy link
Contributor

hop-dev commented Jul 22, 2024

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

kibana-ci commented Jul 22, 2024 •
edited
Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @machadoum

@machadoum machadoum merged commit 240d988 into elastic:main Jul 22, 2024
26 checks passed
@kibanamachine kibanamachine added v8.16.0 backport:skip This commit does not require backporting labels Jul 22, 2024
@tommyers-elastic tommyers-elastic added the Feature:EEM Elastic Entity Model label Jul 23, 2024
@klacabane klacabane mentioned this pull request Jul 25, 2024
Merged
klacabane added a commit that referenced this pull request Jul 26, 2024
@klacabane @kibanamachine
[eem] narrow down index patterns in definition templates (#189182)
10bfb4b 
In #188410 we moved history and
latest index templates from global scope to definition scope. The
definition-scoped templates have a wide pattern that would grep any
other definition template already installed and throw the following
error because of conflicting priority. This change narrows down the
index patterns defined in the templates to only grep the ones from the
installed definition

```
{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": """[illegal_argument_exception
	Root causes:
		illegal_argument_exception: index template [entities_v1_history_admin-console-services_index_template] has index patterns [.entities.v1.history.*] matching patterns from existing templates [entities_v1_history_builtin_services_from_ecs_data_index_template] with patterns (entities_v1_history_builtin_services_from_ecs_data_index_template => [.entities.v1.history.*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority]: index template [entities_v1_history_admin-console-services_index_template] has index patterns [.entities.v1.history.*] matching patterns from existing templates [entities_v1_history_builtin_services_from_ecs_data_index_template] with patterns (entities_v1_history_builtin_services_from_ecs_data_index_template => [.entities.v1.history.*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority"""
}
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@klacabane klacabane klacabane approved these changes

@hop-dev hop-dev hop-dev approved these changes

Assignees

@machadoum machadoum

Labels
backport:skip This commit does not require backporting ci:project-deploy-observability Create an Observability project Feature:EEM Elastic Entity Model release_note:skip Skip the PR/issue when compiling release notes Team:Entity Analytics Security Entity Analytics Team v8.16.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@machadoum @elasticmachine @klacabane @hop-dev @kibana-ci @kibanamachine @tommyers-elastic