Alternate languages for admin names

[missinglink]

It looks like English tokens are not available in the elasticsearch admin indices for locations where the default language is not English.

For instance, the query dionysiou areopagitou, athens will fail with strict admin matching but succeed for the query dionysiou areopagitou, Αθήνα.

It's only by completely ignoring the admin token that the query can succeed.

Note: The language service kicks in afterwards and renders the English token, making it a little confusing to debug.

I think it's important that we start importing more aliases for the terms, which would have to be provided by wof-admin-lookup during PIP.

Some examples:

• dionysiou areopagitou, athens
• عمرو إبن العاص, cairo, egypt

[orangejulius]

That make sense. I was thinking about pelias/schema#349 yesterday, and wondering how much space we would save if we were only indexing, but not storing, admin fields for autocomplete.

Since Elasticsearch has to duplicate all the admin values many times, we aren't getting much "bang for our storage buck", and as we have discovered there is a performance cost to having more data in Elasticsearch: it means fewer records can fit into disk cache.

If we started excluding all the admin fields from the _source field, I suspect our Elasticsearch index size would go down considerably. We are already calling Placeholder on essentially every request anyway. Placeholder is very fast and also can store those admin fields efficiently, so it's a good model.

[orangejulius]

One slight modification to our current behavior that might be useful. Currently, the language middleware in API defaults to English if no language is set via HTTP headers or the lang param. This effectively means we will ask Placeholder for the English name if none is available. Is it possible to calculate a default language value in Placeholder? This would allow us to retain the same behavior as we currently have by going to Elasticsearch.