settings.js overwrites ES default # of shards,replicas

[easherma]

https://github.com/pelias/schema/blob/master/settings.js#L289

This is an important issue to either adjust and/or note in the installation docs. Per #6 , it is nice to be able to override ES default settings as part of the create index process, especially because helps to avoid having to make config changes in multiple places.

However, this default override is an unpleasant surprise for users setting up their own instance. In particular, setting up a single shard is a poor default, and since increasing the number of shards can only done via re-index and import. I'd suggest making the default something more sensible and noting in the install docs that create_index.js already overwrites these settings.

Tangential points:

• It would be great to expand on this and include other typically changed elasticsearch configs. In particular, if heap size could be adjusted here that would be great (realizing that may not be possible or ideal)
• I think it would be a more user-friendly process is create_index (and potentially drop as well, but less important) also created a snapshot of the initialized db pre import data. This is good practice anyways to give the user a fallback point, but is probably overlooked often. (snapshots pre and post import for each of the modules would also be great, but obviously would take more effort).

[orangejulius]

Good catch. I propose we do two things

1.) move the Elasticsearch settings in that block (# of replicas, # of shards, and index_concurrency) into pelias/config as defaults. We point to the defaults there much more frequently, so they'll be a bit more obvious than being buried here. Personally I associate master/settings.js in pelias/schema more with all our analyzer and token filter configuration.

2.) Pick a new default for the number of shards. Elasticsearch uses 5, so maybe there's no reason to think we're smarter than the Elasticsearch people? @missinglink any thoughts on how many shards we should use?

[orangejulius]

Hey @easherma,
We've now moved the general Elasticsearch configuration settings to pelias/config, and there's a new section in the documentation with some suggestions on shard settings. Take a look, let me know if it makes sense, and if the Elasticsearch settings can be easily changed for you.

[orangejulius]

I believe we fixed this in the work in the pull requests connected above my last comment. If there's anything more to do here don't hesitate to let us know in this issue or a new issue. But since we think we've taken care of it, we're closing this issue to help us keep track of things.