Description for the p-values

[Jenny060399]

First of all, many thanks for the R package "oolong"! I have a brief question about the description of the p-values in the Overview vignette. It says that for the Word Intrusion Test the null hypothesis "H0: MP is not better than 1/ n_top_terms" is assumed. However, random guessing would correspond to "1/(number of candidates) = 1/(n_top_terms + 1)" (because of an intruder), if I understand it correctly. Am I misunderstanding something here?

In addition, I would have another question regarding the number of topics to be evaluated for the word intrusion test. I read in another issue that it is critical not to evaluate all topics of the model in the Word Intrusion Test, but in my specific use case it was necessary because I was only interested in the seeded topics of keyATM and seededLDA. I made a change to the code locally for this purpose. However, I am unsure whether I also need to make changes for the statistical tests, i.e. do they use the number of questions or the number of topics of the model as the number of attempts "n"? If you could help me here, I would be very grateful!

[chainsawriot]

@Jenny060399

You are right that the null hypothesis is the number of choices, i.e. n_top_terms + 1.

https://github.com/chainsawriot/oolong/blob/55bb0173b005288e635c937586d26bfa9b7e2bad/R/oolong_summary_tm.R#L43-L59

I will update the overview accordingly.

Statistically, reducing the K (for whatever reason) should not affect the p-value.