-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move pre-configured "keyword" tokenizer to the analysis-common module #24863
Move pre-configured "keyword" tokenizer to the analysis-common module #24863
Conversation
@jpountz, can you have a look at this one? |
Does it mean we can't get normalization to work if analysis-common is not on the classpath? |
Yes. |
The change looks correct to me, but I'm a bit confused about how integration tests that rely on normalization work with this change? We don't put analysis-common on the classpath when running integ tests, do we? |
I suspect we don't actually test custom normalizers without analysis-common installed. We have unit tests but they don't need run this particular code. I could only find some documentation that builds a custom normalizer. I didn't see anything in the rest-api-spec. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK I got it now, existing tests that make sure we apply normalization are done on predefined analyzers such as the standard
analyzer, this is why it keeps working. Sorry for the confusion. LGTM.
Thanks for reviewing! I'll see about removing the conflicts later today. |
Thanks for reviewing @jpountz! |
Moves the
keyword
tokenizer to the analysis-common module. Thekeyword
tokenizer is special because it is used byCustomNormalizerProvider
so I pulled it out into its own PR. To get the move to work I've reworked the lookup from static to one using theAnalysisRegistry
. This seems safe enough.Part of #23658.