discover sources & layers by querying elasticsearch #1316
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
orangejulius
requested changes
Jun 17, 2019
There was a problem hiding this comment.
Nice, this will be really helpful, as we've seen from issues such as pelias/csv-importer#26 and pelias/csv-importer#36 that updating the list of layers is a decent challenge and easy to forget, creating a poor experience.
This was referenced Jun 18, 2019
orangejulius
added a commit
to pelias/dashboard
that referenced
this pull request
Jul 3, 2019
This project previously depended on Elasticsearch types in order to provide a count of the number of records in each layer. Elasticsearch types will be going away in Elasticsearch 6, and are already deprecated in ES5. Instead of relying on types, the count of records per layer is now created using a [terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) similar to the source and layers [autodetection code](pelias/api#1316) recently added to pelias/api. This is one of the last few roadblocks for dropping Pelias's use of types completely and merging PRs like pelias/schema#293 that will drastically simplify our schema. Connects pelias/pelias#461 Connects pelias/pelias#719
orangejulius
added a commit
to pelias/dashboard
that referenced
this pull request
Jul 3, 2019
This project previously depended on Elasticsearch types in order to provide a count of the number of records in each layer. Elasticsearch types will be going away in Elasticsearch 6, and are already deprecated in ES5. Instead of relying on types, the count of records per layer is now created using a [terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) similar to the source and layers [autodetection code](pelias/api#1316) recently added to pelias/api. This is one of the last few roadblocks for dropping Pelias's use of types completely and merging PRs like pelias/schema#293 that will drastically simplify our schema. Connects pelias/pelias#461 Connects pelias/pelias#719
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a feature I started a while back but never quite finished.
It allows users to set api.targets.auto_discover to
truein order to query a list ofsourcesandlayersfrom elasticsearch when the api starts up.It works by sending an aggregation to elasticsearch, this query can take some time (up to several seconds for a full planet build) and so isn't recommended for a production build. (which is why it's disabled by default).
It is, however, very useful for clients who use a lot of custom layer names and forget to update the list manually.
I caught an example of this today from a client, combined with the
address_layer_filtercan cause these custom layers to be excluded from search results when we were only intending to filteraddressdocuments.It's probably worth enabling this for all docker projects by default and explicitly disabling it in production environments where either the mapping is well known ahead of time, or where startup performance is sensitive.