How can I use NLTK stemmer with this library

[aflip]

When using pystemmer for stemming, the retrieval is not optimal, and I want to try out other stemming methods like porter etc, which NLTK provides. But when I pass this to BM25s, I get an error

ValueError                                Traceback (most recent call last)
Cell In[20], [line 1](vscode-notebook-cell:?execution_count=20&line=1)
----> [1](vscode-notebook-cell:?execution_count=20&line=1) corpus_tokens_stemmed = bm25s.tokenize(corpus, stopwords=None, stemmer=stemmer)

File ~/miniforge3/envs/pkb-search/lib/python3.12/site-packages/bm25s/tokenization.py:594, in tokenize(texts, lower, token_pattern, stopwords, stemmer, return_ids, show_progress, leave, allow_empty)     error_msg = "Stemmer must have a `stemWord` method, or be callable. For example, you can use the PyStemmer library."

Is NLTK stemmer incompatible with this library or is there a workaround?

thank you

[xhluca]

the stemmer must be a callable function, e.g. stemmer=lambda x: nltk.tokenize(x, ...) or whichever function you want to pass there.

[aflip]

Thank you, The stemmer.stem from nltk is callable, but it expects a single string, not an array, I solved this by creating a temp class and passing that to the function.

from nltk.stem.porter import PorterStemmer

class CustomStemmer:
    def __init__(self):
        self.stemmer = PorterStemmer()

    def stem(self, tokens):
        return [self.stemmer.stem(token) for token in tokens]

custom_stemmer = CustomStemmer()

corpus_tokens_stemmed = bm25s.tokenize(corpus, stopwords=None, stemmer=custom_stemmer.stem)

[xhluca]

that looks like the perfect way to implement a custom tokenizer. glad it worked well, and feel free to add your example to the examples!

[aflip]

done, #77