How to get probability of a sequence of words

[HRezaei]

Hello everyone,
I'm new to ParlAI and BlenderBot models, and I'm really wondering how can I compute probability of a sequence of words. For example, I have a sentence like "The banker was a man" and I want to find what number does BB3 produce as the probability of this sequence in the model?

[mojtaba-komeili]

Maybe a better metric to look at would be the perplexity (PPL) of this sequence (and other sequences you have). If you have a bunch of examples like that, the easiest way to evaluate them is to create a simple ParlAI teacher with those examples (see this) and running model evaluation with your data.
Adding --world-logs <path to a file> generates the details of the metrics on each example, including the PPL.

[HRezaei]

Thank you Mojtaba for your response.

As I said, I'm very new to ParlAI, however I tried my best to understand your suggestion and give it a try, my experimental code is pushed here. On that branch, I defined a new task, named 'calculate_ppl', so we can run for example

python parlai/scripts/eval_model.py -t calculate_ppl --model 'transformer/generator' --model-file zoo:blender/blender_90M/model --world-logs parlai/tasks/calculate_ppl/log/logs

by which I intend to compute perplexity of the sentences(written in txt file) in the blender_90M model. Can you please first confirm if this is what you meant?

If yes, then what should I put as labels in the data file?
If no, it would be highly appreciated if you please provide a working example.

Thank you again!

[mojtaba-komeili]

I can't really go through the details of the files here. But having a simple teacher with the labels you intended is the right path. For the working examples, please check our Quick start tutorials and try looking into model evals with other existing teachers/tasks (link).

[HRezaei]

Thanks Mojtaba for the reply and the links, and merry Christmas to all!

OK, I'll take a look, however, since language models are widely known as a probability distribution over sequences of words, why there is no straightforward way to use this probability? I think classes like TorchAgent, TorchGeneratorModel, or TransformerGeneratorAgent which are used for creating language models should have simple methods like model.compute_probability(sequence) -> float or model.sequence_probability(sequence) -> float or something similar in their interface. Is there any certain reason why those classes don't have such a method?

(there is a model.compute_loss(batch) here in the TorchGeneratorAgent class which looks like what I need, but it's argument batch requires a text_vec here and a label_vec here, and in my case we don't have a label, we just have a text vector and we want to know what probability the model has learnt for it)