More flexible token metadata logging

[c-flaherty]

Patch description

This is a followup PR to #4169. In that PR, I added support to log token probabilities and token ranks for outputs of ParlAI models. However, after using it, it became clear that we would like to log additional token-level metadata, such as top 10 tokens and top ranked token (relevant for sampling-based decoding methods).

Rather, than add these features directly, I am instead making the token-level metadata object more flexible. In this PR, each token has associated with it a typed dictionary _PathSelectionTokenDetails that contains the token_score and token_rank of the relevant token. No code outside the TreeSearch:select_paths method implementations and this typed dictionaries' definition make any reference to the specific fields in this dictionary. This makes it easy to override this dictionary's definition and a TreeSearch:select_paths implementation to add more verbose metadata. Since different research use-cases may want to generate different data token-level metadata, this approach will be more future-proof.

Additionally, I make a small change to how token probabilities are logged in nucleus sampling. Instead of logging token probs from the truncated (nucleus) distribution, we will now log token probas from the non-truncated distribution.

Finally, I also return normalized probabilities instead of logprobs for token-level probabilities.

Testing steps
pytest tests/test_tga.py

parlai dm --model-file zoo:unittest/transformer_generator2/model --truncate 1024 -v --task integration_tests:multiturn_nocandidate -ne 1 --inference beam --beam-size 3

[emilydinan]

nice @c-flaherty! this will make this code much more flexible!!!

high level comment -- can we call token_scores token_probs throughout?

also flagging for @stephenroller to take a look before merging

[stephenroller]

Lgtm