Enable avoiding mmap bootstrap check #32421

jasontedor · 2018-07-27T02:22:02Z

The maximum map count boostrap check can be a hindrance to users that do not own the underlying platform on which they are executing Elasticsearch. This is because addressing it requires tuning the kernel and a platform provider might now allow this, especially on shared infrastructure. However, this bootstrap check is not needed if mmapfs is not in use. Today we do not have a way for the user to communicate that they are not going to use mmapfs. This commit therefore adds a setting that enables the user to disallow mmapfs. When mmapfs is disallowed, the maximum map count bootstrap check is not enforced. Additionally, we fallback to a different default index store and prevent the explicit use of mmapfs for an index.

Closes #32267

The maximum map count boostrap check can be a hindrance to users that do not own the underlying platform on which they are executing Elasticsearch. This is because addressing it requires tuning the kernel and a platform provider might now allow this, especially on shared infrastructure. However, this bootstrap check is not needed if mmapfs is not in use. Today we do not have a way for the user to communicate that they are not going to use mmapfs. This commit therefore adds a setting that enables the user to disallow mmapfs. When mmapfs is disallowed, the maximum map count bootstrap check is not enforced. This change presents an interesting problem though. The default store type is mmapfs. If a user does not specify a store type, then mmapfs is used. This means that a user should create all indices with a specific store type, say niofs. Normally we would tell such users to add a global index template that specifies a top-level match and sets index.store.type. Now we run into a problem though if a user is using a plugin that adds templates on startup (X-Pack is an example of this). These templates do not (and will not) specify an index.store.type. This means that they will fail template validation because we create a dummy index service to ensure the template is valid. However, we do not inherit other templates during validation so even if the user has a global template that specifies a default index.store.type, this will not be picked up during template validation. To get around this, we do something unusual for us which is add a node-level setting that controls a default index setting. Namely, we add here node.default_index_store_type. It is a smell that we are doing something that is unusual for us. If we had proper template validation, we would not need to do this. An alternative would be to have template validation skip validating index.store.type via a hack. This felt too hacky to me so I elected to take the default index store type route although I am open to other ideas.

elasticmachine · 2018-07-27T02:22:04Z

Pinging @elastic/es-core-infra

* master: Fix ordering of bootstrap checks in docs (elastic#32417) [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints [TEST] Mute failing testConvertLongHexError bump lucene version after backport Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (elastic#32390) [Kerberos] Avoid vagrant update on precommit (elastic#32416)

* master: Remove reference to non-existent store type (elastic#32418) [TEST] Mute failing FlushIT test

jpountz

The change looks good overall but I have some (minor yet existing) reluctance to solve this problem with two settings rather than 1, such as something like node.store.allow_mmapfs. If you like the 2-settings way better then let's do it, I mostly wanted to make sure that you considered doing it with one setting as well.

jibbe · 2018-08-02T18:48:32Z

@jasontedor @jpountz what about just having the single node.default_index_store_type config and make the mmap check remove mmap from the inbuilt list if it doesn't pass? this would auto configure that way.

jasontedor · 2018-08-03T10:55:29Z

@jibbe An issue with only changing the default via a setting is that it would still allow someone to explicitly set the store type on indices to mmapfs which means that we would still want to enforce the bootstrap check.

jibbe · 2018-08-05T23:04:10Z

@jasontedor that makes sense!

…e-types * elastic/master: (199 commits) Watcher: Remove unused hipchat render method (elastic#32211) Watcher: Remove extraneous auth classes (elastic#32300) Watcher: migrate PagerDuty v1 events API to v2 API (elastic#32285) [TEST] Select free port for Minio (elastic#32837) MINOR: Remove `IndexTemplateFilter` (elastic#32841) Core: Add java time version of rounding classes (elastic#32641) Aggregations/HL Rest client fix: missing scores (elastic#32774) HLRC: Add Delete License API (elastic#32586) INGEST: Create Index Before Pipeline Execute (elastic#32786) Fix NOOP bulk updates (elastic#32819) Remove client connections from TcpTransport (elastic#31886) Increase logging testRetentionPolicyChangeDuringRecovery AwaitsFix case-functions.sql-spec Mute security-cli tests in FIPS JVM (elastic#32812) SCRIPTING: Support BucketAggScript return null (elastic#32811) Unmute WildFly tests in FIPS JVM (elastic#32814) [TEST] Force a stop to save rollup state before continuing (elastic#32787) [test] disable packaging tests for suse boxes Mute IndicesRequestIT#testBulk [ML][DOCS] Refer to rules feature as custom rules (elastic#32785) ...

jasontedor · 2018-08-14T19:07:50Z

@jpountz I have taken your suggestion and updated the PR. Would you review?

jasontedor · 2018-08-14T19:19:27Z

@elasticmachine test this please

…e-types * elastic/master: (89 commits) Fix assertion in AbstractSimpleTransportTestCase (elastic#32991) [DOC] Splits role mapping APIs into separate pages (elastic#32797) HLRC: ML Close Job (elastic#32943) Generalize remote license checker (elastic#32971) Trim translog when safe commit advanced (elastic#32967) Fix an inaccuracy in the dynamic templates documentation. (elastic#32890) Logging: Use settings when building daemon threads (elastic#32751) All Translog inner closes should happen after tragedy exception is set (elastic#32674) HLREST: AwaitsFix ML Test Pass DiscoveryNode to initiateChannel (elastic#32958) Add mzn and dz to unsupported locales (elastic#32957) Use settings from the context in BootstrapChecks (elastic#32908) Update docs for node specifications (elastic#30468) HLRC: Forbid all Elasticsearch logging infra (elastic#32784) Only configure publishing if it's applied externally (elastic#32351) Fixes libs:dissect when in eclipse Protect ScriptedMetricIT test cases against failures on 0-doc shards (elastic#32959) (elastic#32968) [Kerberos] Add documentation for Kerberos realm (elastic#32662) Watcher: Properly find next valid date in cron expressions (elastic#32734) Fix some small issues in the getting started docs (elastic#30346) ...

jpountz

Do we still need the node.allowed_index_store_types setting?

jpountz · 2018-08-21T07:52:50Z

server/src/main/java/org/elasticsearch/index/IndexModule.java

+    public static final Setting<Boolean> NODE_STORE_ALLOW_MMAPFS =
+            Setting.boolSetting("node.store.allow_mmapfs", true, Property.NodeScope);
+
+    public static final Setting<List<String>> NODE_ALLOWED_INDEX_STORE_TYPES_SETTING =


I was expecting that this setting would go away?

Oh, that is leftover. It is indeed not used anywhere. Sorry for that. I will remove shortly.

@jpountz I pushed bb9f1f2.

jpountz · 2018-08-21T07:53:51Z

server/src/main/java/org/elasticsearch/indices/IndicesService.java

@@ -228,6 +228,14 @@ public void onRemoval(ShardId shardId, String fieldName, boolean wasEvicted, lon
        this.cacheCleaner = new CacheCleaner(indicesFieldDataCache, indicesRequestCache,  logger, threadPool, this.cleanInterval);
        this.metaStateService = metaStateService;
        this.engineFactoryProviders = engineFactoryProviders;
+
+        // do not allow any plugin-provided index store type to conflict with a built-in type


jpountz

Thank you @jasontedor !

The maximum map count boostrap check can be a hindrance to users that do not own the underlying platform on which they are executing Elasticsearch. This is because addressing it requires tuning the kernel and a platform provider might now allow this, especially on shared infrastructure. However, this bootstrap check is not needed if mmapfs is not in use. Today we do not have a way for the user to communicate that they are not going to use mmapfs. This commit therefore adds a setting that enables the user to disallow mmapfs. When mmapfs is disallowed, the maximum map count bootstrap check is not enforced. Additionally, we fallback to a different default index store and prevent the explicit use of mmapfs for an index.

jasontedor · 2018-08-21T15:28:05Z

Thanks a lot for reviewing @jpountz.

rubin55 · 2018-11-08T17:31:15Z

@jasontedor is this feature in (released) Elasticsearch 6.4.3?

jasontedor · 2018-11-08T17:36:57Z

@rubin55 No, we intend that it will be available in 6.5.0, but that is subject to change (please check the release notes when 6.5.0 is available). Anticipating your next question: sorry, but we do not provide any indication of release dates. I know that can be frustrating, but I hope that you understand. 😇

rubin55 · 2018-11-08T17:44:58Z

@jasontedor Alright, are builds made available of pre-release versions for testing purposes? Since the change is merged to master and 6.5 branch (I think) I would expect a build of master or the 6.5 branch to contain this, would be nice to test!

jasontedor · 2018-11-08T18:22:18Z

@rubin55 You can use a nightly snapshot: use the Docker image docker.elastic.co/elasticsearch/elasticsearch:6.5.0-SNAPSHOT that includes this functionality. Thank you for your interest, and I wish for successful testing!

jasontedor added >enhancement review :Core/Infra/Core Core issues without another label v7.0.0 v6.5.0 labels Jul 27, 2018

jasontedor requested a review from jpountz July 27, 2018 02:22

This was referenced Jul 27, 2018

Fix ordering of bootstrap checks in docs #32417

Merged

Remove reference to non-existent store type #32418

Merged

jasontedor added 3 commits July 27, 2018 08:56

Fix tests

4f74cd0

Merge branch 'master' into allowed-index-store-types

38e64c2

* master: Remove reference to non-existent store type (elastic#32418) [TEST] Mute failing FlushIT test

jpountz reviewed Jul 27, 2018

View reviewed changes

jasontedor added 3 commits August 14, 2018 13:44

Change to one setting

162483e

Readability

14b0d68

jasontedor added 2 commits August 14, 2018 15:13

Fix docs

e600e36

More docs cleanup

6d36f73

jasontedor requested a review from jpountz August 21, 2018 01:00

jpountz reviewed Aug 21, 2018

View reviewed changes

Remove dead setting

bb9f1f2

jasontedor requested a review from jpountz August 21, 2018 10:52

Formatting

ad68c88

jasontedor added 2 commits August 21, 2018 06:53

Formatting

bd5ad6f

Fix comment

2f78a97

jpountz approved these changes Aug 21, 2018

View reviewed changes

jasontedor merged commit bdfcc32 into elastic:master Aug 21, 2018

jasontedor deleted the allowed-index-store-types branch August 21, 2018 15:27

danielmitterdorfer mentioned this pull request Nov 9, 2018

vm.max_map_count bootcheck should be conditional on the use of index.store.type=mmapfs #35377

Closed

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

jasontedor mentioned this pull request Aug 21, 2019

Allow max_map_count=65536 on single nodes for Docker containers #45815

Closed

jpountz mentioned this pull request Nov 11, 2019

Introduce Lucene-based metadata persistence #48733

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable avoiding mmap bootstrap check #32421

Enable avoiding mmap bootstrap check #32421

jasontedor commented Jul 27, 2018 •

edited

Loading

elasticmachine commented Jul 27, 2018

jpountz left a comment

jibbe commented Aug 2, 2018

jasontedor commented Aug 3, 2018

jibbe commented Aug 5, 2018

jasontedor commented Aug 14, 2018

jasontedor commented Aug 14, 2018

jpountz left a comment

jpountz Aug 21, 2018

jasontedor Aug 21, 2018

jasontedor Aug 21, 2018

jpountz Aug 21, 2018

jpountz left a comment

jasontedor commented Aug 21, 2018

rubin55 commented Nov 8, 2018

jasontedor commented Nov 8, 2018

rubin55 commented Nov 8, 2018

jasontedor commented Nov 8, 2018 •

edited

Loading

Enable avoiding mmap bootstrap check #32421

Enable avoiding mmap bootstrap check #32421

Conversation

jasontedor commented Jul 27, 2018 • edited Loading

elasticmachine commented Jul 27, 2018

jpountz left a comment

Choose a reason for hiding this comment

jibbe commented Aug 2, 2018

jasontedor commented Aug 3, 2018

jibbe commented Aug 5, 2018

jasontedor commented Aug 14, 2018

jasontedor commented Aug 14, 2018

jpountz left a comment

Choose a reason for hiding this comment

jpountz Aug 21, 2018

Choose a reason for hiding this comment

jasontedor Aug 21, 2018

Choose a reason for hiding this comment

jasontedor Aug 21, 2018

Choose a reason for hiding this comment

jpountz Aug 21, 2018

Choose a reason for hiding this comment

jpountz left a comment

Choose a reason for hiding this comment

jasontedor commented Aug 21, 2018

rubin55 commented Nov 8, 2018

jasontedor commented Nov 8, 2018

rubin55 commented Nov 8, 2018

jasontedor commented Nov 8, 2018 • edited Loading

jasontedor commented Jul 27, 2018 •

edited

Loading

jasontedor commented Nov 8, 2018 •

edited

Loading