[Reporting] ILM policy for managing reporting indices #100130
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jloleysens
added
(Deprecated) Feature:Reporting
|
@elasticmachine merge upstream |
jloleysens
added
release_note:enhancement
and removed
release_note:skip
|
Pinging @elastic/kibana-app-services (Team:AppServices) |
|
Pinging @elastic/kibana-reporting-services (Team:Reporting Services) |
kobelb
reviewed
May 17, 2021
| phases: { | ||
| hot: { | ||
| actions: {}, | ||
| min_age: '0ms', |
There was a problem hiding this comment.
Do we need the min_age parameter? https://www.elastic.co/guide/en/elasticsearch/reference/master/getting-started-index-lifecycle-management.html#CO461-1 does state that min_age: '0ms' is the default. However, if we can omit this parameter and allow Elastisearch to configure the default, it'll be more future proof if ES changes the default in the future.
There was a problem hiding this comment.
Good point! Thanks for the update!
|
@elasticmachine merge upstream |
jloleysens
commented
May 20, 2021
Comment on lines
+78
to
+88
| body: { | ||
| settings: { | ||
| number_of_shards: 1, | ||
| auto_expand_replicas: '0-1', | ||
| lifecycle: { | ||
| name: this.ilmPolicyName, | ||
| }, | ||
| }, | ||
| mappings: { | ||
| properties: mapping, | ||
| }, |
There was a problem hiding this comment.
Moved this in here to take advantage of the TS typed body. If we want to move the body back outside we can also import the payload type to still have TS checking the payload here.
|
@elasticmachine merge upstream |
tsullivan
approved these changes
May 20, 2021
Comment on lines
163
to
165
| this.logger.debug( | ||
| `Creating ILM policy for managing reporting indices: ${this.ilmPolicyName}` | ||
| ); |
There was a problem hiding this comment.
Suggested change
| this.logger.debug( | |
| `Creating ILM policy for managing reporting indices: ${this.ilmPolicyName}` | |
| ); | |
| this.logger.info(`Creating ILM policy for managing reporting indices: ${this.ilmPolicyName}`); |
|
@elasticmachine merge upstream |
kibanamachine
added a commit
to jloleysens/kibana
that referenced
this pull request
May 25, 2021
* wip; added logic for creating ILM policy at start up * added log when ilm policy is not created * added test for start function * updated ilm policy to not delete data * actually update jest snapshots and remove unused import * updated the ilm policy, removed the min_age for the hot phase * update jest snapshot * removed TODO comment * debug log -> info log Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
kibanamachine
added a commit
to jloleysens/kibana
that referenced
this pull request
May 25, 2021
* wip; added logic for creating ILM policy at start up * added log when ilm policy is not created * added test for start function * updated ilm policy to not delete data * actually update jest snapshots and remove unused import * updated the ilm policy, removed the min_age for the hot phase * update jest snapshot * removed TODO comment * debug log -> info log Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
May 25, 2021
* wip; added logic for creating ILM policy at start up * added log when ilm policy is not created * added test for start function * updated ilm policy to not delete data * actually update jest snapshots and remove unused import * updated the ilm policy, removed the min_age for the hot phase * update jest snapshot * removed TODO comment * debug log -> info log Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/reporting/server/lib/store/store.test.ts # x-pack/plugins/reporting/server/lib/store/store.ts
|
Something went wrong with this merge and GitHub created a commit on elastic/kibana master anyway. Closing manually for now. Not sure whether someone has seen this before? |
jloleysens
added a commit
that referenced
this pull request
May 25, 2021
) * wip; added logic for creating ILM policy at start up * added log when ilm policy is not created * added test for start function * updated ilm policy to not delete data * actually update jest snapshots and remove unused import * updated the ilm policy, removed the min_age for the hot phase * update jest snapshot * removed TODO comment * debug log -> info log Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/reporting/server/lib/store/store.test.ts # x-pack/plugins/reporting/server/lib/store/store.ts
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
May 26, 2021
…deprecation-ilm-policy * 'master' of github.com:elastic/kibana: (101 commits) [ftr] migrate "docTable" service to FtrService class (elastic#100595) [ftr] migrate "listingTable" service to FtrService class (elastic#100606) Fixed comparing real value with formatted according to mode. (elastic#100456) [ftr] migrate "dataGrid" service to FtrService class (elastic#100593) [ftr] migrate "fieldEditor" to FtrService class (elastic#100597) [ftr] migrate "filterBar" service to FtrService class (elastic#100601) [triggersActionsUi] Reduce page load bundle to under 100kB (elastic#97770) [build] Clean jest configs (elastic#100594) refact(NA): remove extra pkg_npm target and add specific target folders for @kbn/analytics on Bazel (elastic#100569) Update dependency @elastic/charts to v29.2.0 (elastic#100587) [Maps] convert LayerPanel to typescript (elastic#100481) [Upgrade Assistant] Address copy feedback (elastic#99632) Open/Closed filter for observability alerts page (elastic#99217) One liner to expose the EQL query for debugging for users (elastic#100565) [KibanaPageLayout] Solution Nav specific styles & props (elastic#100089) [ftr] implement FtrService classes and migrate common services (elastic#99546) [XY] [Lens] Adds opacity slider (elastic#100453) [Reporting] ILM policy for managing reporting indices (elastic#100130) [Reporting] ILM policy for managing reporting indices (elastic#100130) [DOCS] Remove redundant maps attribute (elastic#100426) ... # Conflicts: # x-pack/plugins/reporting/server/lib/store/report_ilm_policy.ts # x-pack/plugins/reporting/server/lib/store/store.test.ts # x-pack/plugins/reporting/server/lib/store/store.ts
9 tasks
💔 Build Failed
Failed CI StepsTest FailuresKibana Pipeline / general / task-queue-process-21 / X-Pack Endpoint API Integration Tests.x-pack/test/security_solution_endpoint_api_int/apis/resolver/entity·ts.Endpoint plugin Resolver tests Resolver tests for the entity route winlogbeat tests "before all" hook for "returns a winlogbeat sysmon event when the event matches the schema correctly"Standard OutStack TraceKibana Pipeline / general / task-queue-process-21 / X-Pack Endpoint API Integration Tests.x-pack/test/security_solution_endpoint_api_int/apis/resolver/entity·ts.Endpoint plugin Resolver tests Resolver tests for the entity route winlogbeat tests "before all" hook for "returns a winlogbeat sysmon event when the event matches the schema correctly"Standard OutStack TraceKibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/management/index_management/indices·js.apis management index management indices list should list all the indices with the expected properties and data enrichersStandard OutStack TraceMetrics [docs]
History
To update your PR or re-run it, just comment with: |
jloleysens
added a commit
that referenced
this pull request
Jun 7, 2021
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Jun 7, 2021
…c#100130)" (elastic#101358) This reverts commit 662fe74. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/reporting/server/lib/store/store.ts
jloleysens
added a commit
that referenced
this pull request
Jun 7, 2021
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Jun 7, 2021
* master: (90 commits) Fix UI breaks on providing long search keyword in 'Search Box' (elastic#101385) Adds css class to EuiDescriptionListDescription in order to break word on exception details card (elastic#101481) [Lens] Increase timings for drag and drop tests (elastic#101380) [Lens] Fix editor react error on configuration panel (elastic#101367) [Fleet] Move integrations to a separate app (elastic#99848) Fix incorrect message displayed on importing Timeline Templates (elastic#101288) [Cases] RBAC (elastic#95058) [APM] Visual improvements for new APM layout with left navigation (elastic#101360) [master] More precise alerts matching (elastic#99820) [Lens] Value in legend (elastic#101353) Revert "[Reporting] ILM policy for managing reporting indices (elastic#100130)" (elastic#101358) [Discover] Fix header row of data grid in Firefox (elastic#101374) Add link to advanced setting in Discover (elastic#101154) Url service locators (elastic#101045) [Timelion] Update the removal message to mention the exact version (elastic#100994) [Security Solution][Detection Engine] Test cases for alias failure test cases where we don't copy aliases correctly (elastic#101437) [Event Log] Adding `type_id` to saved object array in event log (elastic#100939) [Reporting] Add `location.url` info to console message logs (elastic#101427) [Security Solutions][Detection Engine] Fixes timestamp bugs within source indexes when the formats are not ISO8601 format (elastic#101349) Improve Task Manager instrumentation (elastic#99160) ...
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Jun 14, 2021
* wip; added logic for creating ILM policy at start up * added log when ilm policy is not created * added test for start function * updated ilm policy to not delete data * actually update jest snapshots and remove unused import * updated the ilm policy, removed the min_age for the hot phase * update jest snapshot * removed TODO comment * debug log -> info log Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This was referenced Jun 30, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Address points 1 & 2 of #81544
This contribution adds functionality at reporting plugin start time that registers an ILM policy for managing indices.
To do
Release note
Reporting now registers an ILM policy for managing the lifecycle of reports. The policy is
kibana-reportingand the default policy is for reports to live in the hot phase forever. This policy can be updated to manage report indices however the user chooses.Checklist
Delete any items that are not applicable to this PR.