skipping the padding

[mojtaba-komeili]

Patch description
After looking into parlai data_stats crashes that I was running into, I noticed that it was happening when the script was trying to tokenize the non-existing text from a padding message. Added this to skip those messages.
Note: this was happening with chunk teachers, even with batch-size of 1. So the main problem might be from the teacher itself.

Testing steps
My parlai data_stats runs after this.

[stephenroller]

sgtm

[jaseweston]

what is this is_padding() field -- do we always have that?
why not just check if the text we get is an empty string?

[mojtaba-komeili]

If they are using a Message object, instead of dict, that method must be there (link).
But maybe skipping the ones that do not have text field in them covers a wider number of things that could go wrong as well. I change it to that.

[stephenroller]

Image Teachers often give empty text with an image field. We recently changed messages to explicitly mark when they are padding. It's much cleaner.

[mojtaba-komeili]

Regarding the above comments, I keep the skipping condition as before (is_padding) but also set the default value for text to be an empty string, instead of None, in case there are any other messages without text field.