[ILM] Add index_codec to forcemerge action in hot and warm phases #78175

yuliacech · 2020-09-22T16:39:36Z

Summary

Fixes #76907.
This PR adds an index_codec parameter to the forcemerge action in hot and warm phases. The user can either activate this option for higher compression ratio but slower field performance. If left disabled, the default codec is used.
Docs link: https://www.elastic.co/guide/en/elasticsearch/reference/master/ilm-forcemerge.html

Screenshot

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

Release Note

We added an option to select a higher compression codec for force merge action in ILM.

elasticmachine · 2020-09-22T16:49:39Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

jloleysens

Great work @yuliacech ! I tested the code locally and it looks good to me!

I left one comment about adding another test case, but not blocking on that.

jloleysens · 2020-09-23T09:53:56Z

x-pack/plugins/index_lifecycle_management/public/application/services/policies/warm_phase.ts

@@ -84,6 +87,12 @@ export const warmPhaseFromES = (phaseSerialized?: SerializedWarmPhase): WarmPhas
        ? actions.shrink.number_of_shards.toString()
        : '';
    }
+
+    if (actions.set_priority) {


Yikes, I am guessing this was missing before!

jloleysens · 2020-09-23T10:18:42Z

...index_lifecycle_management/public/application/services/policies/policy_serialization.test.ts

@@ -367,4 +368,135 @@ describe('Policy serialization', () => {
    serializePolicy(deserializedPolicy, originalPolicy);
    expect(originalPolicy).toEqual(originalClone);
  });
+
+  test('serialize a policy using "best_compression" codec for forcemerge', () => {


I think it is worth adding a test that asserts our serialisation will remove the "best_compression" setting from a policy that originally has it specified. This will ensure that we do not have a dangling "best_compression" setting.

yuliacech · 2020-09-24T09:48:57Z

Hi @esdocs team, could you please have a look at my wording for the switch label and description for the codec parameter? The docs also mention that the index will be closed and then re-opened if this option is used. Should we put that into UI too?

yuliacech · 2020-09-28T20:44:40Z

@elasticmachine merge upstream

jrodewig

LGTM. I left some suggestions, but the current UI text is good.

Overall, I think we can abstract away things like codec and the compression ratio. Instead, we can focus on the outcome: Better compression of stored fields at the cost of slower performance.

jrodewig · 2020-10-01T12:53:23Z

...index_lifecycle_management/public/application/sections/edit_policy/components/forcemerge.tsx

+const bestCompressionLabel = i18n.translate(
+  'xpack.indexLifecycleMgmt.forcemerge.bestCompressionLabel',
+  {
+    defaultMessage: 'Enable higher compression ratio codec',


Some alternate suggestions in order of preference:

Compress stored fields

Use higher compression

jrodewig · 2020-10-01T13:15:01Z

...index_lifecycle_management/public/application/sections/edit_policy/components/forcemerge.tsx

+                  defaultMessage="Uses DEFLATE for a higher compression ratio but slower stored fields performance
+                  than the default LZ4 codec."


Some alternate suggestions in order of preference:

Better compress stored fields at the cost of slower performance.

Use higher compression for stored fields at the cost of slower performance.

Use the DEFLATE codec for higher compression but slower stored fields performance. If disabled, the default LZ4 codec is used.

yuliacech · 2020-10-05T12:15:25Z

@elasticmachine merge upstream

kibanamachine · 2020-10-06T12:53:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 1d7295b

Metrics [docs]

async chunks size

id	before	after	diff
`indexLifecycleManagement`	231.7KB	233.1KB	+1.5KB

page load bundle size

id	before	after	diff
`indexLifecycleManagement`	89.9KB	89.9KB	+58.0B

History

💚 Build #79574 succeeded c43fdff
💚 Build #77668 succeeded 944b286
💚 Build #76248 succeeded 01493fc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…astic#78175) * [ILM] Add index_codec to forcemerge action in hot and warm phases * [ILM] Fix i18n check errors * [ILM] Add code review suggestions * [ILM] Add code review suggestions Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…8175) (#79652) * [ILM] Add index_codec to forcemerge action in hot and warm phases * [ILM] Fix i18n check errors * [ILM] Add code review suggestions * [ILM] Add code review suggestions Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…into feature/task_manager_429 * 'feature/task_manager_429' of github.com:elastic/kibana: (158 commits) Add license check to direct package upload handler. (#79653) [Ingest Manager] Rename API /api/ingest_manager => /api/fleet (#79193) [Security Solution][Resolver] Simplify CopyableField styling and add comments (#79594) Fine-tunes ML related text on Metrics UI (#79425) [ML] DF Analytics creation wizard: ensure job creation possible when model memory lower than estimate (#79229) Add new "Add Data" tutorials (#77237) Update APM telemetry docs (#79583) Revert "Add support for runtime field types to mappings editor. (#77420)" (#79611) Kibana request headers (#79218) ensure missing indexPattern error is bubbled up to error callout (#79378) Missing space fix (#79585) remove duplicate tab states (#79501) [data.ui] Lazy load UI components in data plugin. (#78889) Add generic type params to search dependency. (#79608) [Ingest Manager] Internal action for policy reassign (#78493) [ILM] Add index_codec to forcemerge action in hot and warm phases (#78175) [Ingest Manager] Update open API spec and add condition to agent upgrade endpoint (#79579) [ML] Hide Data Grid column options when histogram charts are enabled. (#79459) [Telemetry] Synchronous `setup` and `start` methods (#79457) [Observability] Persist time range across apps (#79258) ...

[ILM] Add index_codec to forcemerge action in hot and warm phases

5d40489

yuliacech added Feature:ILM release_note:enhancement v7.10.0 v8.0.0 labels Sep 22, 2020

[ILM] Fix i18n check errors

01493fc

yuliacech requested review from jloleysens and esdocs September 22, 2020 16:49

yuliacech added the Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more label Sep 22, 2020

yuliacech marked this pull request as ready for review September 22, 2020 16:49

yuliacech requested a review from a team as a code owner September 22, 2020 16:49

jloleysens approved these changes Sep 23, 2020

View reviewed changes

Merge branch 'master' into ilm_forcemerge_codec

944b286

jrodewig approved these changes Oct 1, 2020

View reviewed changes

kibanamachine and others added 3 commits October 5, 2020 08:15

Merge branch 'master' into ilm_forcemerge_codec

c43fdff

[ILM] Add code review suggestions

233012b

[ILM] Add code review suggestions

1d7295b

yuliacech merged commit dcb646d into elastic:master Oct 6, 2020

yuliacech mentioned this pull request Oct 6, 2020

[7.x] [ILM] Add index_codec to forcemerge action in hot and warm phases (#78175) #79652

Merged

yuliacech deleted the ilm_forcemerge_codec branch October 7, 2020 12:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ILM] Add index_codec to forcemerge action in hot and warm phases #78175

[ILM] Add index_codec to forcemerge action in hot and warm phases #78175

yuliacech commented Sep 22, 2020 •

edited

Loading

elasticmachine commented Sep 22, 2020

jloleysens left a comment

jloleysens Sep 23, 2020

jloleysens Sep 23, 2020

yuliacech commented Sep 24, 2020

yuliacech commented Sep 28, 2020

jrodewig left a comment

jrodewig Oct 1, 2020

jrodewig Oct 1, 2020

yuliacech commented Oct 5, 2020

kibanamachine commented Oct 6, 2020

		defaultMessage="Uses DEFLATE for a higher compression ratio but slower stored fields performance
		than the default LZ4 codec."

[ILM] Add index_codec to forcemerge action in hot and warm phases #78175

[ILM] Add index_codec to forcemerge action in hot and warm phases #78175

Conversation

yuliacech commented Sep 22, 2020 • edited Loading

Summary

Screenshot

Checklist

Release Note

elasticmachine commented Sep 22, 2020

jloleysens left a comment

Choose a reason for hiding this comment

jloleysens Sep 23, 2020

Choose a reason for hiding this comment

jloleysens Sep 23, 2020

Choose a reason for hiding this comment

yuliacech commented Sep 24, 2020

yuliacech commented Sep 28, 2020

jrodewig left a comment

Choose a reason for hiding this comment

jrodewig Oct 1, 2020

Choose a reason for hiding this comment

jrodewig Oct 1, 2020

Choose a reason for hiding this comment

yuliacech commented Oct 5, 2020

kibanamachine commented Oct 6, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

page load bundle size

History

yuliacech commented Sep 22, 2020 •

edited

Loading