[Response Ops][Alerting] Update common component template generation for framework alerts as data #150384
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ds defined in the package that are not in the mapping
ymao1
force-pushed
the
alerting/faad-resources-common
branch
from
c5b84fc
to
f24a0be
Compare
…ana into alerting/faad-resources-common
ymao1
force-pushed
the
alerting/faad-resources-common
branch
2 times, most recently
from
bc75044
to
761ab42
Compare
ymao1
force-pushed
the
alerting/faad-resources-common
branch
from
761ab42
to
2ca888c
Compare
ymao1
commented
Feb 8, 2023
| }, | ||
| [ALERT_ACTION_GROUP]: { | ||
| type: 'keyword', | ||
| [ALERT_RULE_PARAMETERS]: { |
There was a problem hiding this comment.
We have to keep the flattened mapping for the rule parameters to maintain backwards compatibility
ymao1
commented
Feb 8, 2023
| }, | ||
| [ALERT_ID]: { | ||
| [ALERT_INSTANCE_ID]: { |
There was a problem hiding this comment.
We had considered renaming this to kibana.alert.id but should keep kibana.alert.instance.id for backwards compatibility
ymao1
commented
Feb 8, 2023
| @@ -1,17 +0,0 @@ | |||
| /* | |||
There was a problem hiding this comment.
Moved this definition to the alerting framework
ymao1
commented
Feb 8, 2023
| @@ -1,36 +0,0 @@ | |||
| /* | |||
There was a problem hiding this comment.
Moved this definition to the alerting framework
shahzad31
approved these changes
Feb 22, 2023
marshallmain
approved these changes
Feb 22, 2023
x-pack/plugins/alerting/common/alert_schema/field_maps/mapping_from_field_map.ts
Show resolved
Hide resolved
| @@ -112,16 +110,15 @@ export class ResourceInstaller { | |||
| name: DEFAULT_ALERTS_ILM_POLICY_NAME, | |||
| body: DEFAULT_ALERTS_ILM_POLICY, | |||
| }), | |||
| this.createOrUpdateComponentTemplate({ | |||
There was a problem hiding this comment.
When framework alerts are enabled, the later install functions (installIndexLevelResources and installNamespacedIndexTemplate) that depend on installCommonResources succeeding no longer have the dependency on the ILM policy or ECS templates being installed. As a result, failures or delays in installation of these resources, when handled by the framework, could cause alerts indices to be created incorrectly.
There was a problem hiding this comment.
👍 Thanks for pointing this out! This is a first in a series of PRs to move resource installation to the alerting plugin. When we finish this phase, the feature flag will determine which plugin handles all of the installation, not just pieces of it, so there should not be a case where the framework installs the common component templates but the rule registry is installing index level and namespace level resources.
|
@elasticmachine merge upstream |
There was a problem hiding this comment.
@ymao1 Great job and PR description! 😍
Compare 4 and 7 (ECS component template from rule registry & alerting framework in this branch).
- some new ECS fields for alerting installed template
- each keyword mapped field for alerting installed template should have ignore_above setting
- no kibana.* fields in the alerting installed template
When I was checking 1 and 7 (which I assume is similar to comparing 4 and 7), I didn't see any difference for ignore_above setting (2nd item). Am I missing something?
|
@elasticmachine merge upstream |
|
The ECS template on From your screenshot above, it is showing the Let me know if that makes sense! |
|
@elasticmachine merge upstream |
|
@ymao1 Thanks for the info. You are right, I had an issue with the config, but now it is clear |
|
@elasticmachine merge upstream |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Module Count
Public APIs missing comments
Async chunks
Public APIs missing exports
Page load bundle
Unknown metric groupsAPI count
ESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: cc @ymao1 |
ymao1
added a commit
that referenced
this pull request
Mar 8, 2023
…ponent templates, index templates and concrete write index to framework for alerts-as-data (#151792) Resolves #151697 ## Summary In a previous [PR](#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: #150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
bmorelli25
pushed a commit
to bmorelli25/kibana
that referenced
this pull request
Mar 10, 2023
…for framework alerts as data (elastic#150384) Resolves elastic#150358 ## Summary In a previous [PR](elastic#145581) we started installing a common component template for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for its component templates. In this PR we are doing the following: * Renaming the installed `alerts-common-component-template` to `.alerts-framework-mappings`. * Creating and installing `.alerts-legacy-alert-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup * The combination of the two component templates creates the same set of mappings as the rule registry technical component template * Creating and installing `.alerts-ecs-mappings` component template when `enableFrameworkAlerts: true` on alerting plugin setup (when `enableFrameworkAlerts: false`, the rule registry continues to install this component template * Using the `@kbn/ecs` package provided by core to generate the ECS field map. The rule registry will continue to install the existing ECS field map which is actually a subset of ECS fields * Adding `useLegacy` and `useEcs` flags that allow rule types to specify whether to include the legacy alerts component template and the ECS component template when registering with framework alerts-as-data. * Moved some common functions to alerting framework from the rule registry ## Things to note * When generating the ECS field map, we are now including the `ignore_above` setting from the `@kbn/ecs` package. This changes the ECS component template to include those settings. I tested updating an index with just `"type":"keyword"` mappings to add the `ignore_above` field to the mapping and had no issues so this seems like an additive change to the mapping that will hopefully prevent problems in the future. * The rule registry ECS component template also includes the technical fields which is redundant because the technical component template is automatically installed for all index templates so the framework ECS component template only contains ECS fields. | Previous mapping | Updated mapping | | ----------- | ----------- | | `{ "organization": { "type": "keyword" } }` | `{ "organization": { "type": "keyword", "ignore_above": 1024 } }` | ## To Verify ### Verify that the generated component templates are as expected: Get the following **While running `main`:** 1. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 2. Get the technical component template `GET _component_template/.alerts-technical-mappings` 3. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: false`:** 4. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 5. Get the technical component template `GET _component_template/.alerts-technical-mappings` 6. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true`:** 7. Get the ECS component template `GET _component_template/.alerts-ecs-mappings` 8. Get the technical component template `GET _component_template/.alerts-technical-mappings` 9. Create a detection rule that creates an alert and then get the index mapping for the concrete security alert index `GET .internal.alerts-security.alerts-default-000001/_mapping` 10. Verify that component templates exist for `.alerts-framework-mappings` and `.alerts-legacy-alert-mappings` **Compare the ECS component templates** Compare 1 and 4 (ECS component template from `main` and installed by rule registry in this branch). The difference should be: * no difference in ECS fields * because the rule registry ECS component template also includes technical fields, you will see the 2 new technical fields in this branch Compare 4 and 7 (ECS component template from rule registry & alerting framework in this branch). * some new ECS fields for alerting installed template * each `keyword` mapped field for alerting installed template should have `ignore_above` setting * no `kibana.*` fields in the alerting installed template **Compare the technical component templates** Compare 2 and 5 (technical component template from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 5 and 8 (technical component template from rule registry & alerting framework in this branch). * there should be no difference! **Compare the index mappings** Compare 3 and 6 (index mapping from `main` and installed by rule registry in this branch). The difference should be: * 2 new `kibana.alert` fields (`flapping_history` and `last_detected`) Compare 6 and 9 (index mapping from rule registry & alerting framework in this branch). * some new ECS fields * each `keyword` mapped ECS field should have `ignore_above` setting ### Verify that the generated component templates work with existing rule registry index templates & indices: 1. Run `main` or a previous version and create a rule that uses both ECS component templates & technical component templates (detection rules use both). Let it run a few times. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. Verify that the mapping on the existing `.internal.alerts-security.alerts-default-000001` has been updated to include the latest ECS mappings and the two new technical fields. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
bmorelli25
pushed a commit
to bmorelli25/kibana
that referenced
this pull request
Mar 10, 2023
…ponent templates, index templates and concrete write index to framework for alerts-as-data (elastic#151792) Resolves elastic#151697 ## Summary In a previous [PR](elastic#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: elastic#150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #150358
Summary
In a previous PR we started installing a common component template for framework alerts as data when the
xpack.alerting.enableFrameworkAlertsconfig flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for its component templates.In this PR we are doing the following:
alerts-common-component-templateto.alerts-framework-mappings..alerts-legacy-alert-mappingscomponent template whenenableFrameworkAlerts: trueon alerting plugin setup.alerts-ecs-mappingscomponent template whenenableFrameworkAlerts: trueon alerting plugin setup (whenenableFrameworkAlerts: false, the rule registry continues to install this component template@kbn/ecspackage provided by core to generate the ECS field map. The rule registry will continue to install the existing ECS field map which is actually a subset of ECS fieldsuseLegacyanduseEcsflags that allow rule types to specify whether to include the legacy alerts component template and the ECS component template when registering with framework alerts-as-data.Things to note
ignore_abovesetting from the@kbn/ecspackage. This changes the ECS component template to include those settings. I tested updating an index with just"type":"keyword"mappings to add theignore_abovefield to the mapping and had no issues so this seems like an additive change to the mapping that will hopefully prevent problems in the future.{ "organization": { "type": "keyword" } }{ "organization": { "type": "keyword", "ignore_above": 1024 } }To Verify
Verify that the generated component templates are as expected:
Get the following
While running
main:GET _component_template/.alerts-ecs-mappingsGET _component_template/.alerts-technical-mappingsGET .internal.alerts-security.alerts-default-000001/_mappingWhile running this branch with
xpack.alerting.enableFrameworkAlerts: false:GET _component_template/.alerts-ecs-mappingsGET _component_template/.alerts-technical-mappingsGET .internal.alerts-security.alerts-default-000001/_mappingWhile running this branch with
xpack.alerting.enableFrameworkAlerts: true:GET _component_template/.alerts-ecs-mappingsGET _component_template/.alerts-technical-mappingsGET .internal.alerts-security.alerts-default-000001/_mapping.alerts-framework-mappingsand.alerts-legacy-alert-mappingsCompare the ECS component templates
Compare 1 and 4 (ECS component template from
mainand installed by rule registry in this branch). The difference should be:Compare 4 and 7 (ECS component template from rule registry & alerting framework in this branch).
keywordmapped field for alerting installed template should haveignore_abovesettingkibana.*fields in the alerting installed templateCompare the technical component templates
Compare 2 and 5 (technical component template from
mainand installed by rule registry in this branch). The difference should be:kibana.alertfields (flapping_historyandlast_detected)Compare 5 and 8 (technical component template from rule registry & alerting framework in this branch).
Compare the index mappings
Compare 3 and 6 (index mapping from
mainand installed by rule registry in this branch). The difference should be:kibana.alertfields (flapping_historyandlast_detected)Compare 6 and 9 (index mapping from rule registry & alerting framework in this branch).
keywordmapped ECS field should haveignore_abovesettingVerify that the generated component templates work with existing rule registry index templates & indices:
mainor a previous version and create a rule that uses both ECS component templates & technical component templates (detection rules use both). Let it run a few times.xpack.alerting.enableFrameworkAlerts: falseand verify Kibana starts with no rule registry errors and the rule continues to run as expected.xpack.alerting.enableFrameworkAlerts: trueand verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. Verify that the mapping on the existing.internal.alerts-security.alerts-default-000001has been updated to include the latest ECS mappings and the two new technical fields.Checklist
Delete any items that are not applicable to this PR.