Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calculating log probability over larger document #23

Open
GoogleCodeExporter opened this issue Jul 16, 2015 · 0 comments
Open

Calculating log probability over larger document #23

GoogleCodeExporter opened this issue Jul 16, 2015 · 0 comments

Comments

@GoogleCodeExporter
Copy link

Hello,

I would like to use this LM for classification and therefore I need to 
calculate the log probability of an entire document.

One of the getLogProb() methods state:
"Calculate language model score of an n-gram. <b>Warning:</b> if you
     * pass in an n-gram of length greater than <code>getLmOrder()</code>,
     * this call will silently ignore the extra words of context. In other
     * words, if you pass in a 5-gram (<code>endPos-startPos == 5</code>) to
     * a 3-gram model, it will only score the words from <code>startPos + 2</code>
     * to <code>endPos</code>."

Is it correct to assume that the only way to get the log probability score for 
an entire document (sentence that contains more than LMOrder words) is to split 
up the document in separate n-grams and query the log probability score for 
these separately?

Original issue reported on code.google.com by Dirkcvan...@gmail.com on 26 May 2015 at 7:49

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant