[Security Solution][Detections] Update signals template if outdated and rollover indices #80019

marshallmain · 2020-10-08T16:04:41Z

Summary

This PR updates create_index_route so that it checks if the template needs to be upgraded before creating the index if it doesn't exist. In the case where the index already exists but the template was upgraded, the index rolls over so that the write index has the upgraded mapping.

This will cause a breaking change with old mappings that have risk_score mapped as a keyword in some places. In the current mapping, signal.rule.risk_score is a float so after rolling over there will be a conflict between the old and new signal.rule.risk_score for some features like aggregations.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-10-08T16:05:29Z

Pinging @elastic/siem (Team:SIEM)

spong · 2020-10-12T23:20:25Z

@elasticmachine merge upstream

madirey · 2020-10-13T15:00:58Z

...rity_solution/public/detections/containers/detection_engine/alerts/use_signal_index.test.tsx

@@ -121,6 +125,7 @@ describe('useSignalIndex', () => {
        loading: false,
        signalIndexExists: false,
        signalIndexName: null,
+        signalIndexTemplateOutdated: null,


Should there be a test for when signalIndexTemplateOutdated === true?

…ana into update-signals-template

marshallmain · 2020-10-14T21:56:54Z

jenkins test this

marshallmain · 2020-10-14T23:59:06Z

@elasticmachine merge upstream

kibanamachine · 2020-10-15T01:39:00Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 81bb281

Metrics [docs]

async chunks size

id	before	after	diff
`securitySolution`	10.9MB	10.9MB	+2.0KB

distributable file count

id	before	after	diff
`default`	48494	48495	+1

History

💔 Build #81764 failed 3d6b4e8
💔 Build #81744 failed 3d6b4e8
💚 Build #81214 succeeded 1dd9727
💚 Build #80916 succeeded 0457c69
💔 Build #80889 failed 15e8d6d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…nd rollover indices (elastic#80019) * Modify create_index_route to update template in place if outdated * Update frontend to always call create_index_route * Add template status to GET route * Clean up parameter type * Fix tests and types * Add test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* master: (102 commits) [Resolver] Fix flaky test (elastic#80576) Update Security Solution Bug Report Template (elastic#80668) [Observability] Kibana home page Observability link pointing to `/landing` (elastic#80636) [APM] Update User Experience app callout code to reflect new name (elastic#80641) [APM] Add missing ML privileges (elastic#80553) [DOCS] Adds intro line to the ML plugin readme file (elastic#80631) [ML] Functional tests - fix and re-enable validation API tests (elastic#80617) remove non-existing dependency from uptime plugin (elastic#80623) [ML] Fix job selection flyout overflow (elastic#80621) Move dashboard code in codeowner files to canvas team (elastic#80345) [Security Solution][Detections] Update signals template if outdated and rollover indices (elastic#80019) Sort service list by TPM if health is not shown (elastic#80447) Add in cluster version for sec telemetry sender. (elastic#80545) [Usage Collection] Usage collection add saved objects client to collector fetch context (elastic#80554) Change tag from experimental to beta (elastic#80443) [Metrics UI] Inventory view cleanup (elastic#79881) [Security Solutions][Detection Engine] Critical bug where value lists were not operational (elastic#80368) [Security Solution] Fix networkTopNFlow search strategy response (elastic#80362) [build] Retry docker pull (elastic#80432) add template for Security Solution bugs (elastic#80574) ...

…nd rollover indices (#80019) (#80615) * Modify create_index_route to update template in place if outdated * Update frontend to always call create_index_route * Add template status to GET route * Clean up parameter type * Fix tests and types * Add test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…nd rollover indices (#80019) (#80616) * Modify create_index_route to update template in place if outdated * Update frontend to always call create_index_route * Add template status to GET route * Clean up parameter type * Fix tests and types * Add test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

gchaps · 2020-10-16T15:49:24Z

@marshallmain For this PR to be included in the breaking changes doc, I need a write up with a paragraphs on details and impact. See this breaking changes doc for an example:

marshallmain · 2020-10-26T16:54:09Z

Details: The Security Solution now checks the .siem-signals mapping version and upgrades the mapping if it is outdated when an admin visits the app. This makes new fields that are necessary for new features, such as signal.group.id, available automatically after the upgrade process.

Impact: signal.rule.risk_score was mapped as a keyword prior to 7.9 and was changed to float in 7.9. .siem-signals indices that currently have signal.rule.risk_score mapped as keyword will see mapping conflicts when attempting to aggregate on signal.rule.risk_score.

elasticmachine · 2021-09-22T15:35:35Z

Pinging @elastic/security-solution (Team: SecuritySolution)

marshallmain added 3 commits October 7, 2020 15:27

Modify create_index_route to update template in place if outdated

dbb0b46

Merge branch 'master' into update-signals-template

1146f6b

Update frontend to always call create_index_route

56ca518

marshallmain requested review from a team as code owners October 8, 2020 16:04

marshallmain added v7.10.0 v7.11.0 v8.0.0 Feature:Detection Rules Security Solution rules and Detection Engine release_note:skip Skip the PR/issue when compiling release notes Team:SIEM labels Oct 8, 2020

marshallmain added release_note:breaking and removed release_note:skip Skip the PR/issue when compiling release notes labels Oct 8, 2020

marshallmain added 3 commits October 8, 2020 16:26

Add template status to GET route

1b6907c

Clean up parameter type

15e8d6d

Fix tests and types

0457c69

Merge branch 'master' into update-signals-template

1dd9727

madirey reviewed Oct 13, 2020

View reviewed changes

marshallmain added 2 commits October 14, 2020 15:47

Add test

3490170

Merge branch 'update-signals-template' of github.com:marshallmain/kib…

3d6b4e8

…ana into update-signals-template

dplumlee approved these changes Oct 14, 2020

View reviewed changes

madirey approved these changes Oct 14, 2020

View reviewed changes

Merge branch 'master' into update-signals-template

81bb281

marshallmain merged commit 0c7ca14 into elastic:master Oct 15, 2020

marshallmain deleted the update-signals-template branch October 15, 2020 06:11

marshallmain mentioned this pull request Oct 15, 2020

[7.x] [Security Solution][Detections] Update signals template if outdated and rollover indices (#80019) #80615

Merged

marshallmain mentioned this pull request Oct 15, 2020

[7.10] [Security Solution][Detections] Update signals template if outdated and rollover indices (#80019) #80616

Merged

gchaps mentioned this pull request Nov 3, 2020

[DOCS] 7.10 Release Notes #82380

Merged

1 task

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detections] Update signals template if outdated and rollover indices #80019

[Security Solution][Detections] Update signals template if outdated and rollover indices #80019

marshallmain commented Oct 8, 2020 •

edited

Loading

elasticmachine commented Oct 8, 2020

spong commented Oct 12, 2020

madirey Oct 13, 2020

marshallmain commented Oct 14, 2020

marshallmain commented Oct 14, 2020

kibanamachine commented Oct 15, 2020

gchaps commented Oct 16, 2020

marshallmain commented Oct 26, 2020

elasticmachine commented Sep 22, 2021

[Security Solution][Detections] Update signals template if outdated and rollover indices #80019

[Security Solution][Detections] Update signals template if outdated and rollover indices #80019

Conversation

marshallmain commented Oct 8, 2020 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Oct 8, 2020

spong commented Oct 12, 2020

madirey Oct 13, 2020

Choose a reason for hiding this comment

marshallmain commented Oct 14, 2020

marshallmain commented Oct 14, 2020

kibanamachine commented Oct 15, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

distributable file count

History

gchaps commented Oct 16, 2020

marshallmain commented Oct 26, 2020

elasticmachine commented Sep 22, 2021

marshallmain commented Oct 8, 2020 •

edited

Loading