Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

skipping the padding #3704

Merged
merged 2 commits into from
Jun 16, 2021
Merged

skipping the padding #3704

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c2545d9
skipping the padding
mojtaba-komeili Jun 10, 2021
376eced
empty text
mojtaba-komeili Jun 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion parlai/scripts/data_stats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,14 @@ def keep_token(t):
while not world.epoch_done() and world.total_exs < max_cnt:
world.parley()
act = world.get_acts()[opt.get('agent')]
if act.is_padding():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this is_padding() field -- do we always have that?
why not just check if the text we get is an empty string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they are using a Message object, instead of dict, that method must be there (link).
But maybe skipping the ones that do not have text field in them covers a wider number of things that could go wrong as well. I change it to that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image Teachers often give empty text with an image field. We recently changed messages to explicitly mark when they are padding. It's much cleaner.

continue
for itype in {'input', 'labels'}:
if itype == 'input':
if opt.get('new_line_new_utt'):
txts = act.get('text').split('\n')
else:
txts = [act.get('text')]
txts = [act.get('text', '')]
else:
txts = act.get('labels', act.get('eval_labels', ['']))

Expand Down