Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

autocomplete: extend additional name fields used in multimatch queries #1620

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

missinglink
Copy link
Member

this PR is an experiment with splitting up the name.* fields in order to avoid the negative effects of field norms due to field length, reported in pelias/openstreetmap#507 and better explained in pelias/pelias#862

in particular we see this issue in OSM and WOF due to those sources having more alt names than others, although it applies to all sources.

as discussed on our call today, it might be that pelias/openstreetmap#435 exacerbated the issue (albeit unknown at the time) so reversing that method and moving back to multiple fields using a multi_match query should result in a significant reduction in the effects of the field norms issue on scoring.

although fairly arbitrary, I've identified 4 new fields to begin with:

  • alt - this field will contain all alternative names, so the norms penalty will no longer apply to the primary name. this includes variants, colloquialisms & other alternatives
  • abbr - abbreviations, ie. succulent representations of the primary name
  • code - similar to above but distinct in the case of airports, stop IDs etc.
  • org - brands, operators etc.

we may very well change these, maybe abbr and code can be merged, or org omitted, that's up for discussion.
the main difference is that we attempt to have only a single token indexed per field.

@missinglink
Copy link
Member Author

missinglink commented Mar 31, 2022

I'm not sure if we want to keep using best_fields, maybe cross_fields is better if it doesn't suffer the same norms issue.

@missinglink
Copy link
Member Author

Screenshot 2022-04-01 at 11 40 06

@missinglink
Copy link
Member Author

missinglink commented Apr 4, 2022

this looks very promising:
Screenshot 2022-04-04 at 12 10 52

worth noting we will need to make similar changes to the /v1/search subqueries, otherwise some aliases which were previously searchable are now not (Phoenix Sky Harbor Intern.... in this example, no. 2 on the left)
Screenshot 2022-04-04 at 12 12 39

@missinglink
Copy link
Member Author

missinglink commented Apr 4, 2022

Interestingly, the popularity boosting may now be too strong (rather than too weak as proposed in #1619), or maybe this was always the case 🤔

For example, this /v1/search query has an exact matching result but the scoring of all top n items seems to be heavily influenced by the popularity value:
https://pelias.github.io/compare/#/v1/search?text=pyramids+of+giza&debug=1

Screenshot 2022-04-04 at 12 23 19

orangejulius added a commit to pelias/acceptance-tests that referenced this pull request Apr 5, 2022
These tests have not had great pass rate until
pelias/api#1620, so we didn't know that the
`distanceThresh` value for the coordinate checks weren't quite correct.

Hopefully they will be passing soon!
@missinglink
Copy link
Member Author

missinglink commented Apr 15, 2022

As discussed offline, I've pushed a new commit which changes this behaviour to use wildcards instead of explicit field names, I feel like this is more flexible. The _ delimiter is unfortunately required otherwise German would match the default field. (ie. de* == de && default), using - could potentially cause conflict with hyphenated language codes.

Screenshot 2022-04-15 at 12 18 22

@missinglink
Copy link
Member Author

as-is this PR is safe to merge since it's backward compatible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant